>Dear Rob,
>
> I can see that a common "User Agent: Agent" would help, but how
>could you tell them apart?
I wouldn't want to. I could just block the lot from areas of the server
that I don't want them (accidentally or otherwise) trampling over.
I want my server to be unfriendly to unfriendly agents, robot and busted
proxy servers from Microsoft. Anything that follows my robots.txt won't
be affected.