>How about this instead: Instead of trying to make a HTTP protocal change -
>simply ask all robots to identify themselves as a robot via a (robot) in
>the User-Agent header?
Great idea, but again is begs the question, "What qualifies as a robot?" For
example, does WebCompass (which retrieves two levels deep to summarize
findings) count as a robot? What about other types of "user agent"
interaction?
Getting everyone to agree to put (robot) in their User-Agent will pretty
much require (again) an active definition of what should and shouldn't. In
my opinion, this is even less a cut-and-dried issue than "Should it adhere
to the robot exclusion standard?"
Brian
-------------------------------------------------------------------- Brian Clark - Production Coordinator - bclark@radzone.org ------------------------------------------------------------------ GlobalMedia Design http://www.radzone.org/gmd/ Email Infobot radbot@radzone.org ------------------------------------------------------------------