Think of it in reverse... not all robots give due respect to the /robots.txt
file so if you have a section of tree you want obvious to (people running
graphical) browsers then you can reference it with an server-side clickable
map.
If you *do* want robots to see the tree then you *do* want to include text
links in parallel...
> .... if a site owner
> is allocated say 200MB of transfer (throughput, whatever) for their web
> site, and everyone starts using robots and spiders, then won't these
> robots and spiders *alone* use up the average small web site's transfer
> allocation?
That's a question to ask the provider, I don't suppose many can bundle
individuals' robots.txt files into a general /robots.txt. It's a good
thing to ask however - just to express concern. You ought to get an
offer from the provider to state "if the server is getting hammered
by a particular site we'll block it".
Regards,
Martin Kiff
National Physical Laboratory, UK
mgk@newton.npl.co.uk / mgk@webfeet.co.uk