Brilliantly put, can I quote that :-)
>As the Internet gets bigger and better, I think it is inevitable that
>there will be much greater cooperation between indexers and indexees.
>Stuff such as what has been asked for here *will* be implemented, and
>much more as well, precisely because robots are picking up too
>much garbage using the existing shotgun approach.
Agreed.
>It is probable that this standard will not evolve from the existing
>standard so much the focus of this group, largely because there
>are too many robots and too many sites for any consensus to emerge
>for extending the protocol.
Possible, though I wouldn't discount it completely -- it may be that
this forum does become the vehicle for this, once enough people with
muscle are convinced of the need. I do agree that /robots.txt will likely
not prove suitable for such a mor powerful set of features
>Instead, I think it is likely that some
>large organisation with control over both ends of the process
>(indexer and indexee) will create an internal protocol, generalize
>it to some external organisations with whom it has relationships,
>others will want to participate and begin to implement support, and
>the new "standard" will emerge from there.
Likely indeed. Seen Netscape's Harvest, ehr, catalog server RDM stuff?
-- Martijn
Email: m.koster@webcrawler.com
WWW: http://info.webcrawler.com/mak/mak.html