>On Tue, 12 Nov 1996, Rob Hartill wrote:
>
>>
>> Out of curiosity, how widely used is robots.txt ? Anyone have some
>> figures ?.
>>
>
>Well, look it this way:
>
>Check your log files (access_log) and (error_log) or whatever their names
>on your system. Do:
>grep 'robots.txt' access_log | more and
>grep 'robots.txt' access_error | more
that doesn't tell me how widely used robots.txt is out there, only
what's accessing my site.
..just in case anyone else misread the question.
-- Rob Hartill. Internet Movie Database Ltd. http://www.imdb.com/ _________________________________________________ This messages was sent by the robots mailing list. To unsubscribe, send mail to robots-request@webcrawler.com with the word "unsubscribe" in the body. For more info see http://info.webcrawler.com/mak/projects/robots/robots.html