Re: infoseeks robot is dumb
Otis Gospodnetic (otisg@panther.middlebury.edu)
Fri, 15 Nov 1996 22:13:23 -0500 (EST)
> On 16 Nov 1996, Hrvoje Niksic wrote:
>
> > (k3is@unbsj.ca) wrote:
> > > The question is: if a search engine or a robot doesn't find 'robots.txt'
> > > (return code from HTTP is 404), should it try to request 'robots.txt'
> > > again? Certainly not, but this might make the life of a robot writer
> > > harder!
> >
> > Why? It should not be hard to maintain a hash of hosts on which
> > robots.txt was tried to retrieve.
> >
>
> True....but what if the server administrator get informed about the role
> of 'robots.txt' and s/he decided to create the file, robots.txt? Well,
> certainly the robot has to check for that....
sure, but maybe once in every run, not every time a document is requested...
Otis
--
eZines Database - <URL:http://www.dominis.com/Zines/>
eBooks Dominis Bookstore - <URL:http://www.booksite.com/cgi-bin/zines>
_________________________________________________
This messages was sent by the robots mailing list. To unsubscribe, send mail
to robots-request@webcrawler.com with the word "unsubscribe" in the body.
For more info see http://info.webcrawler.com/mak/projects/robots/robots.html