Re: "real-time" spidering by Lycos

Danny Sullivan (calafia@ix.netcom.com)
Sat, 18 Jan 1997 23:00:09 0000


> Does anyone know what's up with that 'real spidering' ?
> Is it just temporarily not working, or do they have it there for
> people to get impressed ?

My experience is that it is hit and miss with Lycos instantly
spidering a web page. If it has successfully gathered your page,
you'll get this on the same page that comes up after you submitted
your URL:

We successfully spidered your page.

followed by the URL and other information. If you don't get this
information, the Lycos spider has failed to pick up the page for some
reason.

Keep in mind that while the page may instantly be spidered, the page
is NOT instantly added to the catalog. No one does this, although
Alta Vista and Infoseek are the best. They will instantly spider a
page and usually have it in the catalog in about two days.

Finally, remember that just because a page is instantly spidered, the
site itself is not instantly crawled. That comes later, sometimes
much later.


Danny Sullivan, Calafia Consulting
"A Webmaster's Guide to Search Engines"
http://calafia.com/webmasters/
_________________________________________________
This messages was sent by the robots mailing list. To unsubscribe, send mail
to robots-request@webcrawler.com with the word "unsubscribe" in the body.
For more info see http://info.webcrawler.com/mak/projects/robots/robots.html