Re: in-document directive to discourage indexing ?

Rob Turk (rturk@austin.ibm.com)
Mon, 24 Jun 1996 15:46:09 -0500


Kevin Hoogheem wrote:
>
> I was wondering if anyone has thought that a mandatory
> section of a New Robot Standard or even if anyone has talked
> to anyone about making a Admin-Email entry a standard
> in webservers.
> Why should a Admin-Email entry be mandatory, well I can think of
> one reason and that is Since most people that
> have written robots have the ablity to tell if a link is bad or doesnt
> work anymore with there robot they could easily send a message to the
> administrator of that website with the url and description of the problem
> they have on that page, It would be a way that some Big search engines
> can also give back to the community and also would open up a new
> door for other robots that could better perform checking of websites.

Generally, most httpd servers have a configuration option for
administrators to provide e-mail addresses.

The problem in reverse for what you're suggesting would be that, for
each bot on the net (soon to be billions and billions served? yikes!) to
dutifully inform every site admin it can about all the little problems
it has on their servers would cut into the response time for
administrators to problems from REAL USERS (real people trying to get
out online and see what's there or -- hee hee -- actually trying to be
productive).

Let's say I'm running a website that your robot doesn't like, but my
employer does. Am I going to change it just for your bot? What about
the next bot with a problem? It may not like this or that about my
site. Do I care? Goodness no! If your bot has a problem with my site,
I don't want to hear about it unless you're a potential customer, and
even if you are a potential customer I'd rather hear from you (the human
user) than get fifty crappo little messages from your agent.

In short, each bot should report its results to only those who have
requested the bots to search the net in the first place. They could
maybe share info with one another, such as "That website at
255.255.255.255 sure wasn't ebullient. I got stuck in a image map file
of some ladies' shoefitters anonymous cybermall for 15 whole cycles.
Fortunately, my User has blessed me with a prompt timeout
property...don't go there...they only have grey suede pumps in stock
anyhow!" or something to that effect. To their masters, they should be
as concise as possible, to the point of saying "Well, I tried to find
you a great deal on blue suede Doc Martens, but the website at
blah.blah.blah.blah, where I've heard that Suede shoes abound, wasn't as
forthcoming with the goods as we would hope. Perhaps you could e-mail
their administrator at admin@blah.blah.blah.blah." and leave it at
that. That way the user could see for themselves what the goods at that
site (with a web browser of all things), and leave the admin to do
important admin-like stuff.

The shoe emporium people couldn't care less, especially their web admin.
Other bots probably wont care either.

Please don't make it so hapless webmasters have to respond to each and
every one of your programs problems with their sites. That seems to be
a serious breach of netiquette to me, and I only admin a few sites.
Good luck to the guys with lots of sites if ideas like this one find
cachet with the bot developers...

PS. to the guy that sent the last note I got. Did you know that PCMCIA
originally stood for "People Cant Memorize Computer Industry Acronyms?"
(I read that in some magazine or other) Please at least give me a URL
to find out what that alphabet soup of yours was all about... I'm
amazed by the ratio of acronyms to grammatically correct sentences in
your note.

Not completely acronym compliant, but trying...

-- 
Rob Turk <mailto:rturk@austin.ibm.com> Unofficially Speaking.
Have you noticed that all you need to grow healthy, vigorous grass is a
crack in your sidewalk?