Re: who/what uses robots.txt

Erik Selberg (selberg@cs.washington.edu)
21 Nov 1996 14:40:03 -0800

> True. However, if you have a restrictive definition of what
> constitutes a "robot", and create a spec for that, there is nothing
> to stop anyone to define a "foo", and decide that a "foo" should
> adopt and follow the "robots" standard.

Of course that leaves things that an author considers a "bar" an out
not to follow a standard, even if other folks consider it a "foo."

-Erik

-- 
				Erik Selberg
"I get by with a little help	selberg@cs.washington.edu
 from my friends."		http://www.cs.washington.edu/homes/selberg
_________________________________________________
This messages was sent by the robots mailing list. To unsubscribe, send mail
to robots-request@webcrawler.com with the word "unsubscribe" in the body.
For more info see http://info.webcrawler.com/mak/projects/robots/robots.html