I do just that. Most of the robots I'm aware of get a "Forbidden" message
when they try to access stuff I don't want them to. Microsoft's broken proxy
server gets a "Forbidden" for everything on my servers. Access rights are
a good idea.
So, I ask Robert to tell us how we setup access-rights to prevent ActiveAgent
grabbing our email addresses. Would others be interested in having a way
to define (World - ActiveAgent) access-rights on their servers ?. Robert
says we should use access-rights but hasn't provided a reasonable way
to do it AFAIK. He only suggests the all or nothing approach - remove all
mailto: links or accept automated spamming.
Identification via the USER-AGENT header would provide me with a way to
setup access-rights. If you add "email-collector" or similar then the
solution becomes generic and not specific to ActiveAgent.
rob
_________________________________________________
This messages was sent by the robots mailing list. To unsubscribe, send mail
to robots-request@webcrawler.com with the word "unsubscribe" in the body.
For more info see http://info.webcrawler.com/mak/projects/robots/robots.html