Re: who/what uses robots.txt
Erik Selberg (selberg@cs.washington.edu)
22 Nov 1996 13:48:28 -0800
> > Taking ActiveAgent as an example, on a 28800 baud modem,
>
> 28800 MODEM, not "T-1 or higher connectivity".
>
> This is the distinction that's important (at least in my
> opinion). There's a world of difference between a "robot"
> searching from a fat-ass system with a fat-pipe interface,
> and a "agent" crawling from some guy's PC dialed-in PPP,
> SLIP, or whatever.
True, but any agent that may run in a 288 world will probably also be
run in an ISDN or faster world. Thus, you can't rely on agents running
with slow immediate pipes.
-Erik
_________________________________________________
This messages was sent by the robots mailing list. To unsubscribe, send mail
to robots-request@webcrawler.com with the word "unsubscribe" in the body.
For more info see http://info.webcrawler.com/mak/projects/robots/robots.html