This suggests that before we add all sorts of improvements to the standard we
should try to educate the webmasters: it does not matter whether we have 36
options on how often to refetch the damn file if nobody uses it and robots
still fall down holes, wander in test areas, gobble up access logs...
Seriously, how about some sort of concerted effort to educate webmasters
everywhere that with two minutes of their time they can make everyone's life
better: less visits to their site, less junk in indexes (indices?).
--Louis