Re: indexing via redirectors

Sigfrid Lundberg (siglun@gungner.ub2.lu.se)
Tue, 21 Jan 1997 10:27:31 +0100 (MET)


> My question is: What do spiders usually do when they come across such a
> link?
> Will they follow the redirection? If yes, which URL is then indexed?
> This seems to be important for commercial purposes (pay-per-hits). If
> the final (destination) URL is indexed, my program will be useless.
>

If a spider follows redirects at all, it will delete your URL, and index the
destination. That's the only sensible way to handle redirects in general,
isn't it?. Your URL does not deliver any content. This is indeed one of
the problems with the OCLC PURL schemes.

Sigfrid

PS

People do the most crazy things with redirects noadays:

gungner:~$ HEAD -S http://www.sb.gov.se/robots.txt
HEAD http://www.sb.gov.se/robots.txt --> 302 Moved Temporarily
HEAD http://www.sb.gov.se/ --> 200 OK
Date: Tue, 21 Jan 1997 09:22:01 GMT
Server: Netscape-Enterprise/2.0a
Content-Type: text/html
Client-Date: Tue, 21 Jan 1997 09:22:42 GMT

_________________________________________________
This messages was sent by the robots mailing list. To unsubscribe, send mail
to robots-request@webcrawler.com with the word "unsubscribe" in the body.
For more info see http://info.webcrawler.com/mak/projects/robots/robots.html