Using this system, a robot that correctly identifies itself as such will
not be able to accidentally stray into forbidden regions of the server
(well, they won't have much luck if they do, and won't cause damage).
Adding an apache module to the distribution would make more web admins
aware of robots.txt and the issues relating to it. Being the leader, Apache
can implement this and the rest of the pack will follow.
This is a good idea. Some of the control of robot operation is migrated
from the robot operator to the server operator. An Apache module could be
intelligent about high hit rates, even for robots operating within their
designated space, and retrun an appropriate error response.
Skip Montanaro | Today's Acronym: YAUWWWI (yow'-ee) Yet Another Useless
skip@calendar.com | World-Wide Web Idea
(518)372-5583 | --- Check out Musi-Cal: http://concerts.calendar.com/