I have not carefully studied norobots.pl or robots deny text. But it
seems to me that this very fine script relies on the robot operator properly
identifying his robot with user-agent, referrer etc., and not changing the
name of the robot. If someone wanted to hammer a site, they would only have
to make some changes to the robot, and wreak havoc on your server.
As you know, an errant 'bot can do as much damage or more to a server
as a fping floodping attack, and I am very concerned about the lack of
protection available to websites against your robots. There are utilities
and other techniques widely available to stop fping and other internet
nuisances, but I know of very few defenses against errant 'bots. Has anyone
seen any defenses against 'bots?
Thank you in advance for your frank criticism, your ideas,
and your flames.