All major search engines do use it.
> It *could* be that all these elaborate proposals for changes to robots.txt
> will not achieve much, and it *could* have the opposite effect if the
> syntax serves only to scare more people away from using them.
>
> I'm not saying the proposals are a bad idea, just that care needs to be
> taken to keep the syntax simple enough so as not to scare people off.
Good point, the current robots.txt doesn't cover some requirements that have
been brought up here, but with a well-organized document name space it's a
pretty good means of controlling robots access.
A full-featured version of robots.txt V2 might be much harder to implement
and thus not get widely supported (or, people will mess up their robots.txt
configuration :-))
> BTW, I tried to submit a URL to Lycos a couple of times in the last
> month or so, and was told by their automated system that the robots.txt
> prevented them accessing the URL. I checked the robots.txt and it didn't
> prevent access. I sent a bug report to them - no answer, no change. Now
> I wonder if they share the same code with their robot. If they do it's
> probably making all sorts of wrong decisions.
I noticed UltraSeek was case-insensitive, thus blocking /ausbildung also
blocked /Ausbildung. They responded within a few hours and said they would work
on a fix.
Responsiveness with most search engine maintainers generally seems to be low,
though.
Klaus Johannes Rusch
-- e8726057@student.tuwien.ac.at, KlausRusch@atmedia.net http://www.atmedia.net/KlausRusch/ _________________________________________________ This messages was sent by the robots mailing list. To unsubscribe, send mail to robots-request@webcrawler.com with the word "unsubscribe" in the body. For more info see http://info.webcrawler.com/mak/projects/robots/robots.html