It is generally accepted in modern society that one should not enter into
someone elses home without permission. A web server, by its very nature,
invites public access.
>Do any others feel as I do that control over use of my
>information is my responsibility and mine alone? That the
>assumption should be not to index a site that has not explicitly
>given permission to be indexed? (I don't expect much agreement
>here, to be honest. But I thought I would ask.)
It is unfortunate that the robots.txt standard supports exclusion and not
permission. The HTTP standard should have had indexing permission built
right into it such that all servers would support some type of call that
tells robots where they are allowed to go. This could have made it
necessary for permission and denial to be set up during server
configuration.
--------------------
Larry Burke
AKTIV Software
Victoria, B.C.
email: lburke@aktiv.com
web: www.aktiv.com
phone: 604.383.4195