This sounds like a very good idea to me. This could be implemented
by continuing the current format of robots.txt, which allows fine
control for specific robots, but replacing the User-Agent: * with a
User-Agent: MIME type. There would need to be a number of mime types
along the line of robot/indexing, robot/archiving, robot/spamming...
but it shouldn't be too hard to implement. Presumably if robots
belong to more than one type then they should either obey the most
restrictive set of general rules, or limit their activities to those
allowed for that page.
Regards,
Nick Dearnaley,
AutoNomy.
autonomy@stjohns.co.uk