Use of robots.txt to "check status"?

Ed Costello (costello@netcom.com)
Sun, 29 Sep 1996 10:47:47 -0400 (EDT)

Does anyone know of any HTTP/0.9 agents which fetch robots.txt as a
way of checking the status of a given site? My site is getting 1000s
of requests per day for robots.txt and most don't appear to be from
spiders (eg, I'll see the same site come in ever 5 minutes and fetch
the file). I know this seems to be a petty thing to be concerned
about but it ends up skewing traffic stats and makes it difficult to
determine which sites are really "walking" the web, and which sites
are just misusing the robot protocol.
-- 
//name	dd							 -ed costello
//email	dd						  costello@netcom.com