Re: robots.txt , authors of robots , webmasters ....

Ed Carp @ TSSUN5 (ecarp@tssun5.dsccc.com)
Thu, 18 Jan 1996 14:27:35 -0600


> It should be noted that there is a fairly strong case to be
> made that a robot threshing through a non-published web site is
> an illegal activity under the abuse of computing facilities
> statute in U.S. law.

I doubt it. In the first place, if you put up a web server on a well-known
port, there isn't a DA in this country that will support a proscecution
based on this, even if the site isn't "published". First of all, if you
don't want the site accessed on that port, it's *your* responsibility to
protect it. That's why we have login and password programs - if you don't
have a modicum of protection on your site, the courts will take a very dim
view of you trying to get someone nailed.

Doing probes on other ports ("twisting the knobs", as it's called) to see
"what's out there" is generally considered to be an unfriendly act, though.

I think it's patently absurd to suggest that robots by default have no right
to access your pages - if you don't want anyone looking at your pages, why
put up a web site, if not just for your own self-aggrandizment?