This is not true. The problem is that people aren't noticing the work that
*is* being done in the area. I've posted this URL previously, but since
there are still messages being posted (several) that appear to ignore the
proposed standards for meta-information and document-level robot
instructions, here it is again:
http://www.w3.org/pub/WWW/Search/9605-Indexing-Workshop/
Anyone seriously interested in the subject of this list should review the
proposals.
Nick