Checking Log files
Cees Hek (hekc@phoenix.cis.mcmaster.ca)
Thu, 14 Dec 1995 14:32:43 -0600 (CST)
Does anyone have a small script that will parse a log file (NCSA 1.3
common log format) and check for "nasty" robots. I don't have a
/robots.txt file on the server, since we welcome anyone to index our
site, but I would like to keep track of any robots that are hammering the
system.
Currently our log file grows at about a half a Meg a day, and I don't have
time to go through it myself. Any help would be appreciated
Cees Hek
Computing & Information Services Email: hekc@mcmaster.ca
McMaster University
Hamilton, Ontario, Canada