>
> Out of curiosity, how widely used is robots.txt ? Anyone have some
> figures ?.
>
Well, look it this way:
Check your log files (access_log) and (error_log) or whatever their names
on your system. Do:
grep 'robots.txt' access_log | more and
grep 'robots.txt' access_error | more
>From the access_log you will see that lots of search engines such as
infoseek and lycos...etc are looking for this file.....if it is not there
then a line or two are inserted in the error_log file.
So what is my point.....well, even if WebMasters don't care about
'robots.txt', they should at least create an empty file called robots.txt
so that if a search engine requests it no error would be written to the
error_log file!
Regards,
Qusay H. Mahmoud
_________________________________________________
This messages was sent by the robots mailing list. To unsubscribe, send mail
to robots-request@webcrawler.com with the word "unsubscribe" in the body.
For more info see http://info.webcrawler.com/mak/projects/robots/robots.html