Re: robots.txt

David M Banes (dbanes@ozemail.com.au)
Thu, 21 Nov 1996 12:53:16 +1100

HipCrime wrote:
>
> Erik Selberg <selberg@cs.washington.edu> wrote:
>
> > Therefore, it is unclear to me if robots.txt is appropriate;
>
> And this is a very important point. For anything to ever be
> considered as a standard (much less actually BE one), it must
> define clearly what is and is not included. Many client-agent
> programs are appearing, and it's my opinion that most of them
> are mis-classified as "robots".
>
> Maybe you should call this the "agents.txt" file, develop a
> standard for client-driven, user-controlled agents. This is
> distinctly different from the concept of a "robot" (i.e. NOT
> human-controlled).
>
> ... Robert

As a newcomer, and from the discussions I have observed here, I think I
agree with Robert on this one. My bot is in his 'client-driven, user
controlled agents' catagory. It's just an addon for a browser. So
perhaps there should be two simple standards, robots.txt and agents.txt,
rather than one attempting to be both.

On the other hand, it does fall within the general description of a
robot, so maybe after all one standard will suffice. Perhaps call the
file 'access.txt', which any type of software should obey.

David Banes.
[--------------------------------------------------------------------]
email:dbanes@ozemail.com.au CServe - 100446.102@compuserve.com
http://www.ozemail.com.au/~dbanes MSN - banes@msn.com
[--------------------------------------------------------------------]

_________________________________________________
This messages was sent by the robots mailing list. To unsubscribe, send mail
to robots-request@webcrawler.com with the word "unsubscribe" in the body.
For more info see http://info.webcrawler.com/mak/projects/robots/robots.html