Re: Does this count as a robot?
Benjamin Franz (snowhare@netimages.com)
Mon, 8 Jan 1996 08:09:24 -0800 (PST)
On Sun, 7 Jan 1996, Thomas Stets wrote:
> Here is the basic functionality:
>
> - The program starts at a given URL and follows all links that
> are in the same directory or below. (Starting with http://x/a/b/c/...
> it would follow /a/b/c/d/... but not /a/b/e/...)
> (except for IMG graphics)
> - It will, optionally, follow links to other servers one level deep.
> - No links with .../cgi-bin/... or ?parameters are followed.
> - Only http: links are followed.
> - No Document is requested twice. (To prevent loops)
> - It will identify itself with User-agent: and From:
> - It will use HEAD requests when refreshing pages.