>
> Oh, and CGI scripts will be able to detect the "Robot" header
> and barf before they do in real work.
>
> Hmmm, why didn't anyone think of this before?
How about this instead: Instead of trying to make a HTTP protocal change -
simply ask all robots to identify themselves as a robot via a (robot) in
the User-Agent header? This has the huge advantage of not breaking things
at the HTTP protocal level or requiring agreement from the HTTP-WG or
requiring changes to servers. It still has some disadvantages as I noted
in my other message that simply relate to robots identifying themselves
as such in the first place.
-- Benjamin Franz