we want so and so bot, and don't go here or there!
why don't we put in the standard a line
that says.
index-request: 02/20/1996 2 http://www.servonet.com/new-link.html
index-request: 06/31/1996 3 http://www.servonet.com/
index-request: last_time_updated go_2_levels_deep start_indexing_here!
which tells bots where we would like them to index.
- daniel