I've just joined the Robots list, so this is a general 'hello'.
> 'Visit' would allow URLs to be listed for retrieval, and nothing more.
> So you could do:
>
> | Disallow: /
> | Visit: /welcome.html
> | Visit: /products.html
> | Visit: /keywords-and-overview-for-robots.html
>
> Which would be kinda cool and simple, but doesn't scale well to many
> URL's, or to more meta data.
I'd use this for a
Visit: /changes.html
which contains a 'w3new' type list of all pages (all pages I want indexed)
on the server with the most recently modified at the top... crawlers can
do what they like with the information but at least it is there. It might
help however to qualify the 'Visit' keyword in some way to say that
the information is ordered.
Regards,
Martin Kiff
mgk@newton.npl.co.uk