Re: Server name in /robots.txt

Martijn Koster (m.koster@webcrawler.com)
Mon, 22 Jan 1996 05:59:21 -0700


>While I am typing, a hopefully trivial question. Who actually reads the
>HTTP-EQUIV? The documentation I've read doesn't say whether it should
>be understood by the httpd daemon and handed out in response to a HEAD
>request (I haven't noticed my, maybe elderly copies of the, CERN or
>NCSA daemons actually doing this)

The server is supposed to parse the document, and slam the value into
an HTTP header. This is of course a waste of server CPU and bandwidth
for the majority of cases, and opens a whole can of worms with the
semantics of HTTP header namespace collisions.

It isn't widely implemented.

> or should it be understood by the
>browser or robot as it GETs the page?

This makes far more sense -- let user-agents decide what they want to do
with the data; parsing it out of a HTTP stream is simple, and a browser
could even drop the connection afer reading the field if that's all it
needed.

So the idea is that you can do both HTTP-EQUIV=foo and NAME=bar in the
same META tag. The last draft I saw on the subject had HTTP-EQUIV
as the main thing, with NAME being optional. I think it makes far
more sense to have NAME, and abolish HTTP-EQUIV, or at least make
it a secondary choice.

In fact it'd be good if robots started to promote this. I'd add it
to WebCrawler if I wasn't buried in other work...

-- Martijn

Email: m.koster@webcrawler.com
WWW: http://info.webcrawler.com/mak/mak.html