> I am someone who runs one of these problem sites. When I realized
> the mistake I had made in not setting up a proper robots.txt file (after
> I had caused problems, unfortunately), I was happy to put all my
> offending scripts in a separate directory and exclude them. I don't see
> where anyone aware of the problem would ever want any output from these
> types of scripts indexed. Wouldn't there always be a top-level html
> page that would be enough of a reference? Perhaps you could add an
> education step when putting such sites on your s---list, and send the
> site an automated note pointing out the problem?
I think that the specific thing which gives the search engines the value
<http://www.tardis.ed.ac.uk/~mikedlr/biography.html>
Scottish Climbing Archive: <http://www.tardis.ed.ac.uk/~mikedlr/climbing/>
Linux/Unix clone@ftp://src.doc.ic.ac.uk/packages/linux/sunsite.unc-mirror/docs/