Re: How long to cache robots.txt

Daniel T. Martin (MARTIND@carleton.edu)
Thu, 18 Jul 1996 20:44:24 -0500 (CDT)


Ok; I guess that I was more concerned that there was some problem that was
causing your (InfoSeek's) cache to up and die frequently, and so posted to the
list in case others had noticed similar events.

Not really that it would hurt my server much if it were retreived with every
access (except maybe in filling up log space) - I'm beginning to think that
Netscape combined with certain users' graphics-laden web pages is the real
threat to our webserver's performance, and not any slightly impolite 'bot.

Then again, we don't serve out much more than static documents.

Does anyone have a count on the average number of bytes of net traffic
generated by a hit on a user's page (including inline and background images,
java classes, etc.)? Curiosity.

DANIEL