I gather you are wondering why it will find http://www-rfi.eunet.fr/,
but not follow the link on that homepage to http://www.rfi.fr? Most
commercial engines will only follow relative links under the current
URL. Imagine if they didn't and they ran into Yahoo? They'd explode!
If the owner of http://www.rfi.fr would like that site to show up on
Lycos, they'll have to submit the URL.
Of course if a robot did visit http://www.rfi.fr it would probably choke
on the large number of cookies hurled at it. Hmmm, what does happen
when a robot encounters a cookie? Something new to ponder! Thanks!
Thomas Merlin wrote:
>
> Hello again,
>
> I'm sorry to bother you all with my avalanche of questions, but here
> goes...:-)
>
> 1) I've been trying to reference a site http://www.rfi.fr on the main
> search engines, and some of them completely refuse to reference it, like
> Lycos (http://www.lycos.com) or HotBot (http://www.hotbot.com)
>
> Any idea why ?
>
> To find RFI, try typing the followong keywords : radio france
> internationale
>
> 2) If a home page is made out of frames, is it ok to include the meta tags
> KEYWORDS and DESCRIPTION in the file that 'holds' the frames, the one that
> has no other content then the <FRAMESET> and <NOFRAMES> tags...or is there
> a more intelligent way of referencing such a site ?
>
> Thanks a whole lot.
>
> Regards,
>
> Tom.
>
> _________________________________________________
> This messages was sent by the robots mailing list. To unsubscribe, send mail
> to robots-request@webcrawler.com with the word "unsubscribe" in the body.
> For more info see http://info.webcrawler.com/mak/projects/robots/robots.html
-- ================================================================== Mitch Allen WebDroid http://www.webdroid.com P.O. Box 6569 mailto:mitch@webdroid.com Boston, MA. 02114 _________________________________________________ This messages was sent by the robots mailing list. To unsubscribe, send mail to robots-request@webcrawler.com with the word "unsubscribe" in the body. For more info see http://info.webcrawler.com/mak/projects/robots/robots.html