Re: who/what uses robots.txt

HipCrime (HipCrime@HipCrime.com)
Thu, 21 Nov 1996 15:33:34 -0800


Hi Eric ...

This statement expresses the point I'm trying to make:

> Run it from your corporate net with T1 or higher connectivity.

This is NOT the setup that ActiveAgent runs in, and it's a safe
bet that other "agent" authors are also writing code for these
desktop systems (i.e. Pentium, Windows-95, 32MegRAM, etc).

My last note said (sorry everybody for having to repeat this):

> Taking ActiveAgent as an example, on a 28800 baud modem,

28800 MODEM, not "T-1 or higher connectivity".

This is the distinction that's important (at least in my
opinion). There's a world of difference between a "robot"
searching from a fat-ass system with a fat-pipe interface,
and a "agent" crawling from some guy's PC dialed-in PPP,
SLIP, or whatever.

... Robert
_________________________________________________
This messages was sent by the robots mailing list. To unsubscribe, send mail
to robots-request@webcrawler.com with the word "unsubscribe" in the body.
For more info see http://info.webcrawler.com/mak/projects/robots/robots.html