Re: we should help spiders and not say NO!

Daniel Lo (daniel@servonet.com)
Fri, 28 Jun 1996 13:49:25 -0700


The current robot standard says

we want so and so bot, and don't go here or there!

why don't we put in the standard a line
that says.

index-request: 02/20/1996 2 http://www.servonet.com/new-link.html
index-request: 06/31/1996 3 http://www.servonet.com/

index-request: last_time_updated go_2_levels_deep start_indexing_here!

which tells bots where we would like them to index.

- daniel