Re: Broadness of Robots.txt (Re: Washington again !!!)

Erik Selberg (selberg@cs.washington.edu)
21 Nov 1996 09:43:47 -0800

> I totally agree with these statements. I would suggest a slightly
> different implementation. The standard should include a list of general
> behavior classifications and assign a fictitious "User-Agent" name to each
> class.
>
> An agent will scan the robots.txt looking for *either* its own specific
> agent name or the appropriate ficititious class name. If it sees either of
> these, it uses that rule set. There is obviously ambiguity in the case
> where both matches are found. Should the burden be placed on the webadmin
> to put the specific rule sets first?

Hmm... rather than using a User-Agent, perhaps using a User-Type
similar to Content-type (someone else made a similar suggestion, now
that I think about it). For example:

User-Type: robot/webcrawler # WebCrawler's robot
User-Type: robot/scooter # DEC's robot
User-Type: watcher/metacrawler # MetaCrawler

Or is this too much headache in terms of trying to add yet another
MIME type to the HTTP standard?

-Erik

-- 
				Erik Selberg
"I get by with a little help	selberg@cs.washington.edu
 from my friends."		http://www.cs.washington.edu/homes/selberg
_________________________________________________
This messages was sent by the robots mailing list. To unsubscribe, send mail
to robots-request@webcrawler.com with the word "unsubscribe" in the body.
For more info see http://info.webcrawler.com/mak/projects/robots/robots.html