Re: Suggestion to help robots and sites coexist a little better

Rob Hartill (robh@imdb.com)
Thu, 18 Jul 1996 17:32:41 -0600 (MDT)


>At 9:04 PM 7/17/96, Rob Hartill wrote:
>
>>before anyone focuses on this, please will robot owners agree a string
>>to add to the USER_AGENT so that server admins can do something with it.
>>This should be a quick and painless task; it just need doing.
>>
>>Some suggestions for strings to use..
>>
>>(robot)
>>(automated)
>>(follows-robots.txt)
>>(server-friendly)
>
>These mean nothing without associated symantics. The last two seem pointless
>(-ly political).

They were added as 'filler' to offer some more ideas for alternatives.

>For robots (using the established autmated recursion
>definition) the string "robot" makes more sense

Okay.

>>I'm no expert on the syntax.. reformat as appropriate,
>
>If you want to propose something, at least take the effort to lookup
>syntactic restraints, rather than leaving that to mailing-list disscusion.
>So much faster and less confusing. Your use of "()" is against the spec,
>because they are tspecials (which aren't allowed in tokens).

You mean to say that
Mozilla/3.0b5 (X11; I; HP-UX A.09.07 9000/715)

is illegal? I'm shocked ;-)

>Section 3.8 of the HTTP/1.1 draft says:
>
> product = token ["/" product-version]
>
>so the string "robot" fits that.
>
>That took the whole of 25 seconds to confirm :-)

I would have checked the spec but at the time my ISP was down
due to "wet lines" and I didn't have a hardcopy.

>>pick your favorite and let us know. Thanks.
>
>I'm not at all convinced this solves any problem, but if it makes you happy...

It can be put to good use immediately.

rob