Yes...so exercise that control and restrict access to the material
that you don't want read. Control over the security of your
site and your data is your responsibility and yours alone. I'm not
saying that robots should TRY to invade your privacy, but your
comparison of your web site and your house is a little off, I think.
>>If I leave my house unlocked, I have given my permission for any and
>>all to come in and read my personal papers. Does this strike anyone
>>else as somewhat absurd?
I think it's more analogous to leaving your office unlocked in a office
building accessible to the public. You don't expect someone to sit down
and read all your stuff, but then again, you don't necessarily expect
that no one will notice what you have lying around your office.
Robots are accustomed to going where they can. The robots.txt file is
as much or more for the robot's benefit as the site's. I tend to agree
with an earlier contributor that many sites don't have a robots.txt,
have no need for one, and can't be expected to have one. If all these
sites are excluded from indexes....
Besides, if you're relying on robots faithfully abiding by whatever you
have in robots.txt for your security scheme, you're only keeping out the
robots (and human browsers) who don't want your private data. Anyone
that wants it can get it, if you don't protect it. I'm not advocating
that, I'm just saying that's the case.
Like it or not, putting info on the Web is publishing. Lack of
advertising doesn't mean something hasn't been published. The failure
of a chapter to appear in the table of contents doesn't mean it's not
in the book.
Again, I don't WANT your privacy invaded, but if you put your stuff in
a public place and don't restrict access to it....
-- John Lammers