> (k3is@unbsj.ca) wrote:
> > The question is: if a search engine or a robot doesn't find 'robots.txt'
> > (return code from HTTP is 404), should it try to request 'robots.txt'
> > again? Certainly not, but this might make the life of a robot writer
> > harder!
>
> Why? It should not be hard to maintain a hash of hosts on which
> robots.txt was tried to retrieve.
>
True....but what if the server administrator get informed about the role
of 'robots.txt' and s/he decided to create the file, robots.txt? Well,
certainly the robot has to check for that....
There seem to be LOTS of unanswered issues about robots....not too many
people know about them and the file 'robot.txt'!!!
Have fun,
Qusay H. Mahmoud
_________________________________________________
This messages was sent by the robots mailing list. To unsubscribe, send mail
to robots-request@webcrawler.com with the word "unsubscribe" in the body.
For more info see http://info.webcrawler.com/mak/projects/robots/robots.html