> I can imagine, when robots (altavista) are querying my page, this will
> result in a generation of a *huge* amound of 'new' URL's on the spot, and
> an indexing of these 'URL's. This is not my problem, of course ;)
Well -it's hoging your CPU time as well as the robots.
> But what is the best method to avoid this? My CGI-scripts trying to
> identify the robot? Use of robots.txt?...
I'd say using robots.txt to disallow access to the URL which
generates the other URL's. People who visit you pages will find the
page in question easily enough- it's an unwritten law :)
Chris, ChegHchu djajVam djajKak!
wanabe robot programer - don't shout at me......to much <g>
-----------------------------------------------------------
chris@jm-crowther.co.uk
www.dungeon.com/~jmcrowther/chris.html