Re: Suggestion to help robots and sites coexist a little better
Randy Terbush (randy@zyzzyva.com)
Mon, 15 Jul 1996 15:15:15 -0500
> Rob Hartill writes:
>
> Here's my suggestion:
>
> ...
>
> Using this system, a robot that correctly identifies itself as such will
> not be able to accidentally stray into forbidden regions of the server
> (well, they won't have much luck if they do, and won't cause damage).
>
> Adding an apache module to the distribution would make more web admins
> aware of robots.txt and the issues relating to it. Being the leader, Apache
> can implement this and the rest of the pack will follow.
>
> This is a good idea. Some of the control of robot operation is migrated
> from the robot operator to the server operator. An Apache module could be
> intelligent about high hit rates, even for robots operating within their
> designated space, and retrun an appropriate error response.
>
> Skip Montanaro | Today's Acronym: YAUWWWI (yow'-ee) Yet Another Useless
> skip@calendar.com | World-Wide Web Idea
> (518)372-5583 | --- Check out Musi-Cal: http://concerts.calendar.com/
I agree. I should think that most robot operators would enjoy
this "more inteligent" connection to the server.