robots.txt buffer question.

Jeremy Sigmon (jsigmon@www.hsc.wvu.edu)
Tue, 22 Oct 1996 19:23:46 -0400 (EDT)

Messages sorted by: [ date ][ thread ][ subject ][ author ]
Next message: Brent Boghosian: "Re: robots.txt buffer question."
Previous message: Jennifer C. O'Brien: "McKinley -- 100% error rate"
In reply to: Jeremy Sigmon: "Re: The Internet Archive robot"
Next in thread: Martijn Koster: "Re: The Internet Archive robot"

Do most people read in a limited size for the robots.txt file?
I was just wondering is someone linked /dev/urandom to robots.txt and
you requested the file would your robot crash?

======================================================================
Jeremy Sigmon B.S. ChE |
Web Developer of the Robert C. Byrd Health | Use
Sciences Center of West Virginia University | FreeBSD
WWW.HSC.WVU.EDU | Now
Graduate Student in Computer Science |
Office : 293-1060 |

Next message: Brent Boghosian: "Re: robots.txt buffer question."
Previous message: Jennifer C. O'Brien: "McKinley -- 100% error rate"
In reply to: Jeremy Sigmon: "Re: The Internet Archive robot"
Next in thread: Martijn Koster: "Re: The Internet Archive robot"