Is the "robots@webcrawler.com" mailing list the "robots-gripes" list or
the list for developers trying to make user agent software better? I
don't know if there is a "real" official robots-gripe list, but such a
distinction could be beneficial. E.g. developer tips, instructions, q&a
etc. could continue on one list, and then people who are pursuing the
perpetrator of some bad bot or just unhappy about user agents in general
could have another list somewhere. It could be "badbots@webcrawler.com"
or "robots-gripes@webcrawler.com"
Of course, please don't flame me if this is a bad idea. It's early here
and I haven't had my cup of coffee yet.
-- Rob
Betsy Dunphy wrote:
>
> What would appear to be a robot just came through my site. To say
> it does not adhere to the Standards for Robot Exclusion is an
> understatement. Picked up files that are expressly forbade in
> /robots.txt - actually never picked up robots.txt. In a 7 minute
> period picked up 88 files. No User-Agent to identify it. All I have
> is this:
>
> 194.97.0.55
>
> As I write this, I find I'm not surprised at the 'antisocial'
> reactions of some Webmasters out there. If mine were not a
> commercial site, I'd be screaming alot more loudly at anyone who
> writes/runs robots such as I've just experienced.
>
> Anyone have any idea who or what this is?
-- Rob Turk <mailto:rturk@austin.ibm.com> Always Un-Officially Speaking Web Work <http://toolbox.austin.ibm.com/~rturk> http://www.megalith.com The one good thing about repeating your mistakes is that you know when to cringe.