Re: More Robot Talk

Theo Van Dinter (felicity@kluge.net)
Fri, 17 Jan 1997 13:39:36 -0500 (EST)

On Fri, 17 Jan 1997, [ISO-8859-1] Jaakko Hyv=E4tti wrote:
> On Fri, 17 Jan 1997, Captain Napalm wrote:
> > Another idea I had (and it was just an idea) was to have my robot sen=
d a
> > message to the web master at a site (if an address can be found - postm=
aster
> > as a last resort) giving the location of the robots.txt specification i=
f a
> > robots.txt file was not found (404), plus a small file they can use:
>=20
> You would end up sending this automated message to most sites you crawl=
,
> and that will sooner or later mean trouble for you.=20

I tend to manually send a message to the webmaster of a site who has an
incorrect robots.txt (like www.att.com did for a while ... <g> Had
"Dissallow: /cgi-bin/" ... (notice the double 's' in disallow)).

since robots.txt isn't required, why send mail about it? (some sites
don't need one either ...)

--
 --------------------------------------------------------------------------=
-
 Theo Van Dinter=09=09=09www: http://www.kluge.net/~felicity/
 (Vice)President WPI Lens and Lights=09      Active Member in SocComm Films
 Member of WPI ACM=09=09=09      AME for the Masque C-Term Show


             If ignorance is bliss, why aren't more people happy?
 --------------------------------------------------------------------------=
-


_________________________________________________
This messages was sent by the robots mailing list. To unsubscribe, send mail
to robots-request@webcrawler.com with the word "unsubscribe" in the body.
For more info see http://info.webcrawler.com/mak/projects/robots/robots.html