Re: indexing via redirectors

Mike Burner (burner@archive.org)
Tue, 21 Jan 1997 10:42:25 -0800

Messages sorted by: [ date ][ thread ][ subject ][ author ]
Next message: Jeff Drost: "Re: indexing via redirectors"
Previous message: Nick Arnett: "Re: Info on large scale spidering?"
Maybe in reply to: Patrick Berchtold: "indexing via redirectors"

At 01:31 PM 1/21/97 -0000, olly@muscat.co.uk wrote:
>
>Is there any reason to follow redirects on robots.txt? Are there any sites
>which have a legitimate robots.txt (rather than something dumb like this)
>which is on a redirect?
>

http://www.americanexpress.com/robots.txt

Mike

_________________________________________________
This messages was sent by the robots mailing list. To unsubscribe, send mail
to robots-request@webcrawler.com with the word "unsubscribe" in the body.
For more info see http://info.webcrawler.com/mak/projects/robots/robots.html

Next message: Jeff Drost: "Re: indexing via redirectors"
Previous message: Nick Arnett: "Re: Info on large scale spidering?"
Maybe in reply to: Patrick Berchtold: "indexing via redirectors"