RE: indexing via redirectors
Martin.Soukup (martin.soukup@fulcrum.com)
Tue, 21 Jan 1997 10:15:51 -0500
>----------
>From: Patrick Berchtold[SMTP:berchtold@www.student.isoe.ch]
>Sent: Monday, January 20, 1997 6:19 PM
>To: Robots Mailing List
>Subject: indexing via redirectors
>
>>Hello
>>
>>I just wrote a tiny CGI program counting unique visits to my sponsors
>>pages. It is called like
>>http://my.domain.com/redirect/redirect.exe/dest.domain.com/path/file.ext
>>
>>
>>It first does some logging and then makes the server send a '302 Moved
>>Temporarily' status code including the new destination URL
>>dest.domain.com/path/file.ext. If your browser follows this link it
>>will finally display this URL.
>>
>>My question is: What do spiders usually do when they come across such a
>>link?
>>Will they follow the redirection? If yes, which URL is then indexed?
>>This seems to be important for commercial purposes (pay-per-hits). If
>>the final (destination) URL is indexed, my program will be useless.
Most spiders will follow the redirection and index the final page. Smart
spiders won't.
_________________________________________________
This messages was sent by the robots mailing list. To unsubscribe, send mail
to robots-request@webcrawler.com with the word "unsubscribe" in the body.
For more info see http://info.webcrawler.com/mak/projects/robots/robots.html