Re: infoseeks robot is dumb

Hrvoje Niksic (hniksic@srce.hr)
16 Nov 1996 02:12:33 +0100

(k3is@unbsj.ca) wrote:
> The question is: if a search engine or a robot doesn't find 'robots.txt'
> (return code from HTTP is 404), should it try to request 'robots.txt'
> again? Certainly not, but this might make the life of a robot writer
> harder!

Why? It should not be hard to maintain a hash of hosts on which
robots.txt was tried to retrieve.

-- 
Hrvoje Niksic <hniksic@srce.hr> | Student at FER Zagreb, Croatia
--------------------------------+--------------------------------
main(){printf(&unix["\021%six\012\0"],(unix)["have"]+"fun"-0x60);}
_________________________________________________
This messages was sent by the robots mailing list. To unsubscribe, send mail
to robots-request@webcrawler.com with the word "unsubscribe" in the body.
For more info see http://info.webcrawler.com/mak/projects/robots/robots.html