Re: Do robots have to follow links ?

Theo Van Dinter (felicity@kluge.net)
Mon, 6 Jan 1997 23:40:48 -0500 (EST)


On Mon, 6 Jan 1997, Thomas Merlin wrote:

> Sorry for this one-time-in-a-life newbie question that has been discussed
> thousands of times on this list, but do robots have to follow links ? I
> mean, will they browse/scan a page that has no link pointing to it ?

The easy answer question I would have is ... Without a link, how would the
spider know that the file existed? (assuming it's not a 'standardly'
named file (like /robots.txt))

--
-----------------------------------------------------------------------------
Theo Van Dinter				 www: http://www.kluge.net/~felicity/
Vice-President WPI Lens and Lights	       Active Member in SocComm Films
Member of WPI ACM			       AME for the Masque C-Term Show

"2 wrongs don't make right but 3 rights make a left." - TVD -----------------------------------------------------------------------------

_________________________________________________ This messages was sent by the robots mailing list. To unsubscribe, send mail to robots-request@webcrawler.com with the word "unsubscribe" in the body. For more info see http://info.webcrawler.com/mak/projects/robots/robots.html