Re: Do robots have to follow links ?

Nick Arnett (narnett@Verity.COM)
Mon, 06 Jan 1997 18:52:21 -0800


At 11:38 PM 1/6/97 +0100, you wrote:
>Hi again,
>
>Sorry for this one-time-in-a-life newbie question that has been discussed
>thousands of times on this list, but do robots have to follow links ? I
>mean, will they browse/scan a page that has no link pointing to it ?
>
>I guess it would be hell if those kinds of pages could be referenced
>because everybody would publish more and more pages full of keywords and
>descriptions, etc.

A robot won't find that sort of page unless its URL is submitted directly to
it; there's no way for it to discover the page.

However, a server that allows directory listings might allow such a document
to be found, if there's a link to the directory listing.

I don't see any reason that the hell induced by such activity would be any
different from the present forms of robot hell... ;-)

Nick

---------------------------------------
Verity Inc.
Connecting People with Information

Product Manager, Advanced Technology
408-542-2164; home office 408-369-1233; fax 408-541-1600
http://www.verity.com

_________________________________________________
This messages was sent by the robots mailing list. To unsubscribe, send mail
to robots-request@webcrawler.com with the word "unsubscribe" in the body.
For more info see http://info.webcrawler.com/mak/projects/robots/robots.html