Re: Suggestion to help robots and sites coexist a little better

Martijn Koster (m.koster@webcrawler.com)
Thu, 18 Jul 1996 12:27:31 -0700


At 9:04 PM 7/17/96, Rob Hartill wrote:

>before anyone focuses on this, please will robot owners agree a string
>to add to the USER_AGENT so that server admins can do something with it.
>This should be a quick and painless task; it just need doing.
>
>Some suggestions for strings to use..
>
>(robot)
>(automated)
>(follows-robots.txt)
>(server-friendly)

These mean nothing without associated symantics. The last two seem pointless
(-ly political). For robots (using the established autmated recursion
definition) the string "robot" makes more sense

>I'm no expert on the syntax.. reformat as appropriate,

If you want to propose something, at least take the effort to lookup
syntactic restraints, rather than leaving that to mailing-list disscusion.
So much faster and less confusing. Your use of "()" is against the spec,
because they are tspecials (which aren't allowed in tokens).

Section 3.8 of the HTTP/1.1 draft says:

product = token ["/" product-version]

so the string "robot" fits that.

That took the whole of 25 seconds to confirm :-)

>pick your favorite and let us know. Thanks.

I'm not at all convinced this solves any problem, but if it makes you happy...

-- Martijn

Email: m.koster@webcrawler.com
WWW: http://info.webcrawler.com/mak/mak.html