>> Finally -- I never understood why robots.txt was exclusion only.
>> Why does it not have some of positive hints added? I.e. you are
>> allowed & welcome to browse XXXX/fred.html. Was this a choice
>> built upon pragmatism -- thinking that this would open a can of
>> worms?
I too would like to see something like this. Or at least some way of
prioritizing pages.
Jaakko> I do not believe it is a problem to give robots URLs, they are
Jaakko> pretty good at finding them themselves.
A little too good sometimes. The problem comes when one has an
archive of something like a manual that is under development. I
maintain two such archives, and had several robots going through the
pages while pieces were being changed actively, while other more
static pages were ignored.
Regards,
-- steve@miranova.com baur