interactive generation of URL's

Fred Melssen (MELSSEN@AZNVX1.AZN.NL)
Mon, 15 Jul 1996 11:03:57 +0100 (MET)

On one of my sites, people can access a database, by way of a perl
cgi/method=post form. I'm adding some search utilities to this script.
The first search is still submitted by a POST method. The results will
contain URL's, (as if they were submitted by a GET method). This means
that in theory, an unlimited amount of URL's can be generated.

I can imagine, when robots (altavista) are querying my page, this will
result in a generation of a *huge* amound of 'new' URL's on the spot, and
an indexing of these 'URL's. This is not my problem, of course ;)

But what is the best method to avoid this? My CGI-scripts trying to
identify the robot? Use of robots.txt?...

Regards,

-fred

------------------------------------------------------------------------
Fred Melssen | Manager Electronic Information Services
P.O.Box 9104 | Centre for Pacific Studies | Phone and fax:
6500 HE Nijmegen | University of Nijmegen | 31-024-378.3666 (home)
The Netherlands | Email: melssen@aznvx1.azn.nl | 31-024-361.1945 (fax)
| http://www.kun.nl/~melssen | PGP key available
------------------------------------------------------------------------