>
>
> > My question is: What do spiders usually do when they come across such a
> > link?
> > Will they follow the redirection? If yes, which URL is then indexed?
> > This seems to be important for commercial purposes (pay-per-hits). If
> > the final (destination) URL is indexed, my program will be useless.
> >
>
> If a spider follows redirects at all, it will delete your URL, and index the
> destination.
I wish. I put a 301 (Moved Permanently) redirect on a page a *year and
half ago* and what I found was that some popular search engines were
indexing the page content *anyway* (as per the HTTP spec, the body had a
hyperlink pointing to the new page). I finally gave up and put in a copy
of the page it redirects to with a '<BASE HREF' with the correct URL.
-- Benjamin Franz_________________________________________________ This messages was sent by the robots mailing list. To unsubscribe, send mail to robots-request@webcrawler.com with the word "unsubscribe" in the body. For more info see http://info.webcrawler.com/mak/projects/robots/robots.html