I hate to sound heretical, but why is everyone so concerned about this '5%'
figure? I'm sure everyone on this list is a sufficiently sophisticated
programmer to have developed the sort of complex web systems that robots.txt
is supposed to protect, but most web servers probably don't need a robots.txt.
Now, 'most' might not be 95% of all web servers, but the problem is still not
as bad as it might seem. We don't *need* to inform every 'webmaster' who
downloads a server kit and can write HTML; only the hackers.
Monier, do you have any guesses on what fraction of servers *should* have a
robots.txt?
-- Mordechai T. Abzug http://umbc.edu/~mabzug1 mabzug1@umbc.edu finger -l mabzug1@gl.umbc.edu Assembly programmers drive stick shifts.