Re: who/what uses robots.txt

Rob Hartill (robh@imdb.com)
Fri, 22 Nov 1996 01:02:00 +0000 (GMT)


HipCrime wrote:

>Taking ActiveAgent as an example, on a 28800 baud modem,
>experience shows that it processes about 1-2 pages/second.
>Any server which can not withstand that, should really be
>considering an upgrade.

Define a "page" !

Since when has any robot/agent been intelligent enough to know
what effort is required to create these "pages" ?

click <here> to download a smiley face in ASCII.
The page containing ":-)" comes back.

click <here> to turn the satelite 0.5 degrees, map the surface of
the earth in a 100 mile radius and image process the results to
determine how many trees have fallen this week...
The page containing "1" comes back.

looks like the smiley face is 3 times as expensive to me. Maybe they
need 10 crays to let you rapid fire that URL at 28800.

rob
_________________________________________________
This messages was sent by the robots mailing list. To unsubscribe, send mail
to robots-request@webcrawler.com with the word "unsubscribe" in the body.
For more info see http://info.webcrawler.com/mak/projects/robots/robots.html