> I gather you are wondering why it will find http://www-rfi.eunet.fr/,
> but not follow the link on that homepage to http://www.rfi.fr? Most
> commercial engines will only follow relative links under the current
> URL. Imagine if they didn't and they ran into Yahoo? They'd explode!
> If the owner of http://www.rfi.fr would like that site to show up on
> Lycos, they'll have to submit the URL.
Sounds like a good concept, but what about the people who don't believe in
relative URLs?
(I suppose my answer would be 'then those people are stupid', along with
the people who change their 'Error 404 - Page not found' error to redirect
to the main page of the server, so there are no 404 errors, the people who
put web pages in their /robots.txt file, etc...)
-- ----------------------------------------------------------------------------- Theo Van Dinter www: http://www.kluge.net/~felicity/ Vice-President WPI Lens and Lights Active Member in SocComm Films Member of WPI ACM AME for the Masque C-Term Show"I just bought a cured ham... Wonder what it had..." - TVD -----------------------------------------------------------------------------
_________________________________________________ This messages was sent by the robots mailing list. To unsubscribe, send mail to robots-request@webcrawler.com with the word "unsubscribe" in the body. For more info see http://info.webcrawler.com/mak/projects/robots/robots.html