Re: indexing via redirectors

olly@muscat.co.uk
21 Jan 1997 13:31:24 -0000

Sigfrid Lundberg <siglun@gungner.ub2.lu.se> writes:
>People do the most crazy things with redirects noadays:
>
>gungner:~$ HEAD -S http://www.sb.gov.se/robots.txt
>HEAD http://www.sb.gov.se/robots.txt --> 302 Moved Temporarily
>HEAD http://www.sb.gov.se/ --> 200 OK
>[etc]

Hmm, looks like it redirects any page which doesn't exist:

$ HEAD -S http://www.sb.gov.se/any.bogus.url
HEAD http://www.sb.gov.se/any.bogus.url --> 302 Moved Temporarily
HEAD http://www.sb.gov.se/ --> 200 OK
Date: Tue, 21 Jan 1997 12:29:14 GMT
Server: Netscape-Enterprise/2.0a
Content-Type: text/html
Client-Date: Tue, 21 Jan 1997 13:24:03 GMT

Which is more than a little confusing for the user, as they don't get any
feedback that the page they asked for isn't there, just a different page
instead.

Is there any reason to follow redirects on robots.txt? Are there any sites
which have a legitimate robots.txt (rather than something dumb like this)
which is on a redirect?

Olly
_________________________________________________
This messages was sent by the robots mailing list. To unsubscribe, send mail
to robots-request@webcrawler.com with the word "unsubscribe" in the body.
For more info see http://info.webcrawler.com/mak/projects/robots/robots.html