Erik raises the point I hit all the time (although comes to the opposite
conclusion) ... a point that serves stressing again as the push moves on for
the next standard.
In short, robots.txt has become a poor bandaid for a big wound. Many people
(from MetaCrawlers to ActiveAgents to whatever Microsoft has up their sleeve
) will continue to see robots.txt as a protocol for indexing robots ...
something that even the RFC does nothing to dispel. While the distinction
between "browser" and "robot" was both obvious and academic, that is no
longer the case.
Perhaps it shouldn't even be robots.txt anymore ... maybe that sends the
wrong message. Maybe nonhuman.txt ... or agent.txt. I certainly wouldn't
want to see some other group fill the vacuum by coming up with an agents.txt