Re: Checking Log files
Mark Schrimsher (mschrimsher@twics.com)
Fri, 15 Dec 1995 11:11:57 +0900
>Does anyone have a small script that will parse a log file (NCSA 1.3
>common log format) and check for "nasty" robots. I don't have a
>/robots.txt file on the server, since we welcome anyone to index our
>site, but I would like to keep track of any robots that are hammering the
>system.
>
>Currently our log file grows at about a half a Meg a day, and I don't have
>time to go through it myself. Any help would be appreciated
>
>
>Cees Hek
>Computing & Information Services Email: hekc@mcmaster.ca
>McMaster University
>Hamilton, Ontario, Canada
You can make a robots.txt file that permits all accesses, and then check
the log for requests for that file. But it won't catch robots that don't
check for robots.txt.
--Mark