> On Sun, 2 Jun 1996, Ann Cantelow wrote:
>
> > I am someone who runs one of these problem sites. When I realized
> > the mistake I had made in not setting up a proper robots.txt file (after
> > I had caused problems, unfortunately), I was happy to put all my
> > offending scripts in a separate directory and exclude them. I don't see
> > where anyone aware of the problem would ever want any output from these
> > types of scripts indexed. Wouldn't there always be a top-level html
> > page that would be enough of a reference? Perhaps you could add an
> > education step when putting such sites on your s---list, and send the
> > site an automated note pointing out the problem?
>
> I think that the specific thing which gives the search engines the value
>
I can't see that- not for this millenium anyway ;) . What if the
Library of Congress decided to make all of their books available online
by having scripts pull them out of a database by a call number that is
passed in the path-info, and build html pages for them? It seems to me
to be out of the scope of a search engine to index all of those books.
In my case there is a growing database of hundreds-soon to be thousands-
of poems. It seems to me the poems themselves don't belong there either.
If the database was small though, you're right, indexing might be
desireable. Maybe that's where the problem lies. We don't know until
we collect it what size of database we're dealing with. Where and how
do we draw the line?
Regards,
Ann Cantelow
cantelow@athena.csdco.com
http://www.csd.net/~cantelow