Re: infoseeks robot is dumb

Hrvoje Niksic (hniksic@srce.hr)
16 Nov 1996 02:34:40 +0100

(k3is@unbsj.ca) wrote:
> On 16 Nov 1996, Hrvoje Niksic wrote:
> > Why? It should not be hard to maintain a hash of hosts on which
> > robots.txt was tried to retrieve.
> >
>
> True....but what if the server administrator get informed about the role
> of 'robots.txt' and s/he decided to create the file, robots.txt? Well,
> certainly the robot has to check for that....

I think it is also no problem to add an expire on the hash-check, but
I see what you are driving at -- all these gotchas *do* complicate a
robot-writer's life.

-- 
Hrvoje Niksic <hniksic@srce.hr> | Student at FER Zagreb, Croatia
--------------------------------+--------------------------------
I'm a Lisp variable -- bind me!
_________________________________________________
This messages was sent by the robots mailing list. To unsubscribe, send mail
to robots-request@webcrawler.com with the word "unsubscribe" in the body.
For more info see http://info.webcrawler.com/mak/projects/robots/robots.html