Do robots have to follow links ?

Ross A. Finlayson (raf@tomco.net)
Mon, 06 Jan 1997 19:50:32 -0500

I think that any server that had a list of references should have a
contentindex.txt list that maybe had this kind of info for each resourc at
the site:

Title
URL
Keywords
Valid Until
Related Index
Refresh Index Date

and any other myriad statistics they wanted to publish about their site.
Then a crawler would never have to make multiplt http hits to get all the
contect of the site as desired by the ontent creator. Part of the server's
algorithm might remove all the duplicate keywords to discourage "flood"
style indexing.

That would seem to help out a lot for any search engine that could get a
bunch of people to use it.

Just some $0.02,

Ross A. Finlayson
raf@tomco.net

_________________________________________________
This messages was sent by the robots mailing list. To unsubscribe, send mail
to robots-request@webcrawler.com with the word "unsubscribe" in the body.
For more info see http://info.webcrawler.com/mak/projects/robots/robots.html