Re: indexing via redirectors

Hrvoje Niksic (hniksic@srce.hr)
21 Jan 1997 14:24:21 +0100

Sigfrid Lundberg (siglun@gungner.ub2.lu.se) wrote:
> > Will they follow the redirection? If yes, which URL is then indexed?
> If a spider follows redirects at all, it will delete your URL, and
> index the destination. That's the only sensible way to handle
> redirects in general, isn't it?.

This looks more like a thing to do after a 301 response to me.
RFC2068 says for 301:

The requested resource has been assigned a new permanent URI and any
future references to this resource SHOULD be done using one of the
returned URIs. Clients with link editing capabilities SHOULD
automatically re-link references to the Request-URI to one or more of
the new references returned by the server, where possible.

For 302 RFC says:

The requested resource resides temporarily under a different URI.
Since the redirection may be altered on occasion, the client SHOULD
continue to use the Request-URI for future requests.

-- 
Hrvoje Niksic <hniksic@srce.hr> | Student at FER Zagreb, Croatia
--------------------------------+--------------------------------
If a President doesn't do it to his wife, he'll do it to his country.
_________________________________________________
This messages was sent by the robots mailing list. To unsubscribe, send mail
to robots-request@webcrawler.com with the word "unsubscribe" in the body.
For more info see http://info.webcrawler.com/mak/projects/robots/robots.html