It *could* be that all these elaborate proposals for changes to robots.txt
will not achieve much, and it *could* have the opposite effect if the
syntax serves only to scare more people away from using them.
I'm not saying the proposals are a bad idea, just that care needs to be
taken to keep the syntax simple enough so as not to scare people off.
-=-=-
BTW, I tried to submit a URL to Lycos a couple of times in the last
month or so, and was told by their automated system that the robots.txt
prevented them accessing the URL. I checked the robots.txt and it didn't
prevent access. I sent a bug report to them - no answer, no change. Now
I wonder if they share the same code with their robot. If they do it's
probably making all sorts of wrong decisions.
-- Rob Hartill. Internet Movie Database Ltd. http://www.imdb.com/