> A reporter interviewed me today about the future of the robots
> exclusion file (robots.txt) and as we talked...
I think I kinda started all this with my little posting of 10/11:
--------------------------------------------------------------------
Is there a way to allow access to a subdirectory, while disallowing
everything else (short of exhausively listing every file)?
E.g, something like:
Disallow: /
Allow: /public/
--------------------------------------------------------------------
That's pretty much like access.conf:
allow from all
deny from big.bad@wolf.com
Now we've gotten to 300,000 web servers sending us all email
every day telling us which of their pages changed yesterday,
NP-complete regex's, etc.
Maybe you full-timers like all this, but it seems WAY too complex
for us simple-minded little robot-runners and webmasters.
My solution, by the way, was just to reopen the door to Scooter
and watch that he behaves himself this time (he was doing cgi-bin
searches, that's why we excluded him in the first place.)
Maybe we could try a little "KISS" for awhile?
Just one guy's opinion. But when my robots folder is threatening
to get bigger than my inbox, I start getting a little worried.
Fred "Don't Wanna UNDESCRIBE (again)" Lenherr