RE: crawling FTP sites

William Dan Terry (wterry@netpubsintl.com)
Wed, 11 Sep 96 16:39:51

Messages sorted by: [ date ][ thread ][ subject ][ author ]
Next message: Ross A. Finlayson: "Re: Public Access Nodes / Copywrited Nodes"
Previous message: Brewster Kahle: "Re: The Internet Archive robot (fwd)"

I'd say that you want to take in a bunch of factors. Connect info that sites give you should
be used to decide what is appropriate. I've been to one site that only allows two anonymous
logins. So treading lightly is in order.

Off the top of my head some factors to consider:

number of allowed anonymous logins
number of current anonymous logins
robot.txt

Peace, William

______________W__i__l__l__i__a__m_____D__a__n_____T__e__r__r__y______________
How do we acquire wisdom along with all these shiny things? (David Brin)

wterry@netpubsintl.com
http://www.netpubsintl.com
Director of Technology, NetPubs International

Next message: Ross A. Finlayson: "Re: Public Access Nodes / Copywrited Nodes"
Previous message: Brewster Kahle: "Re: The Internet Archive robot (fwd)"