another suggestion

Rob Hartill (robh@imdb.com)
Fri, 22 Nov 1996 17:23:05 +0000 (GMT)

Robots should be allowed to ask for multiple copies of the same URL
at one time, so that they don't need to make repeated requests...

qe20.lycos.com - - [18/Nov/1996:16:10:24 -0600] "GET /robots.txt HTTP/1.0" 200 703
qe20.lycos.com - - [18/Nov/1996:16:13:15 -0600] "GET /robots.txt HTTP/1.0" 200 703
qe20.lycos.com - - [18/Nov/1996:16:13:18 -0600] "GET /robots.txt HTTP/1.0" 200 703
qe20.lycos.com - - [18/Nov/1996:16:14:21 -0600] "GET /robots.txt HTTP/1.0" 200 703
qe20.lycos.com - - [18/Nov/1996:16:14:42 -0600] "GET /robots.txt HTTP/1.0" 200 703
qe20.lycos.com - - [18/Nov/1996:16:17:29 -0600] "GET /robots.txt HTTP/1.0" 200 703
qe20.lycos.com - - [18/Nov/1996:16:17:43 -0600] "GET /robots.txt HTTP/1.0" 200 703
qe20.lycos.com - - [18/Nov/1996:16:23:14 -0600] "GET /robots.txt HTTP/1.0" 200 703
qe20.lycos.com - - [18/Nov/1996:16:23:23 -0600] "GET /robots.txt HTTP/1.0" 200 703
qe20.lycos.com - - [18/Nov/1996:16:24:58 -0600] "GET /robots.txt HTTP/1.0" 200 703
qe20.lycos.com - - [18/Nov/1996:16:25:54 -0600] "GET /robots.txt HTTP/1.0" 200 703
qe20.lycos.com - - [18/Nov/1996:16:28:39 -0600] "GET /robots.txt HTTP/1.0" 200 703
qe20.lycos.com - - [18/Nov/1996:16:31:35 -0600] "GET /robots.txt HTTP/1.0" 200 703
qe20.lycos.com - - [18/Nov/1996:16:33:35 -0600] "GET /robots.txt HTTP/1.0" 200 703
qe20.lycos.com - - [18/Nov/1996:16:35:02 -0600] "GET /robots.txt HTTP/1.0" 200 703
qe20.lycos.com - - [18/Nov/1996:16:35:03 -0600] "GET /robots.txt HTTP/1.0" 200 703
qe20.lycos.com - - [18/Nov/1996:16:35:12 -0600] "GET /robots.txt HTTP/1.0" 200 703
qe20.lycos.com - - [18/Nov/1996:16:35:27 -0600] "GET /robots.txt HTTP/1.0" 200 703
qe20.lycos.com - - [18/Nov/1996:16:36:17 -0600] "GET /robots.txt HTTP/1.0" 200 703
qe20.lycos.com - - [18/Nov/1996:16:36:22 -0600] "GET /robots.txt HTTP/1.0" 200 703
qe20.lycos.com - - [18/Nov/1996:16:36:31 -0600] "GET /robots.txt HTTP/1.0" 200 703
qe20.lycos.com - - [18/Nov/1996:16:36:36 -0600] "GET /robots.txt HTTP/1.0" 200 703
qe20.lycos.com - - [18/Nov/1996:16:37:27 -0600] "GET /robots.txt HTTP/1.0" 200 703
qe20.lycos.com - - [18/Nov/1996:16:37:27 -0600] "GET /robots.txt HTTP/1.0" 200 703
qe20.lycos.com - - [18/Nov/1996:16:37:45 -0600] "GET /robots.txt HTTP/1.0" 200 703
qe20.lycos.com - - [18/Nov/1996:16:38:40 -0600] "GET /robots.txt HTTP/1.0" 200 703
qe20.lycos.com - - [18/Nov/1996:16:39:37 -0600] "GET /robots.txt HTTP/1.0" 200 703
qe20.lycos.com - - [18/Nov/1996:16:42:40 -0600] "GET /robots.txt HTTP/1.0" 200 703
qe20.lycos.com - - [18/Nov/1996:16:44:08 -0600] "GET /robots.txt HTTP/1.0" 200 703
qe20.lycos.com - - [18/Nov/1996:16:45:03 -0600] "GET /robots.txt HTTP/1.0" 200 703
qe20.lycos.com - - [18/Nov/1996:16:47:10 -0600] "GET /robots.txt HTTP/1.0" 200 703
qe20.lycos.com - - [18/Nov/1996:16:47:14 -0600] "GET /robots.txt HTTP/1.0" 200 703
qe20.lycos.com - - [18/Nov/1996:16:47:21 -0600] "GET /robots.txt HTTP/1.0" 200 703
qe20.lycos.com - - [18/Nov/1996:16:49:06 -0600] "GET /robots.txt HTTP/1.0" 200 703
qe20.lycos.com - - [18/Nov/1996:16:50:21 -0600] "GET /robots.txt HTTP/1.0" 200 703
qe20.lycos.com - - [18/Nov/1996:16:50:26 -0600] "GET /robots.txt HTTP/1.0" 200 703
qe20.lycos.com - - [18/Nov/1996:16:51:29 -0600] "GET /robots.txt HTTP/1.0" 200 703
qe20.lycos.com - - [18/Nov/1996:16:55:19 -0600] "GET /robots.txt HTTP/1.0" 200 703
qe20.lycos.com - - [18/Nov/1996:16:57:16 -0600] "GET /robots.txt HTTP/1.0" 200 703
qe20.lycos.com - - [18/Nov/1996:16:58:44 -0600] "GET /robots.txt HTTP/1.0" 200 703
qe20.lycos.com - - [18/Nov/1996:17:00:16 -0600] "GET /robots.txt HTTP/1.0" 200 703
qe20.lycos.com - - [18/Nov/1996:17:02:19 -0600] "GET /robots.txt HTTP/1.0" 200 703
qe20.lycos.com - - [18/Nov/1996:17:07:05 -0600] "GET /robots.txt HTTP/1.0" 200 703
qe20.lycos.com - - [18/Nov/1996:17:12:11 -0600] "GET /robots.txt HTTP/1.0" 200 703
qe20.lycos.com - - [18/Nov/1996:17:12:12 -0600] "GET /robots.txt HTTP/1.0" 200 703
qe20.lycos.com - - [18/Nov/1996:17:12:51 -0600] "GET /robots.txt HTTP/1.0" 200 703
qe20.lycos.com - - [18/Nov/1996:17:14:40 -0600] "GET /robots.txt HTTP/1.0" 200 703
qe20.lycos.com - - [18/Nov/1996:17:15:23 -0600] "GET /robots.txt HTTP/1.0" 200 703
qe20.lycos.com - - [18/Nov/1996:17:15:41 -0600] "GET /robots.txt HTTP/1.0" 200 703
qe20.lycos.com - - [18/Nov/1996:17:16:02 -0600] "GET /robots.txt HTTP/1.0" 200 703
qe20.lycos.com - - [18/Nov/1996:17:16:08 -0600] "GET /robots.txt HTTP/1.0" 200 703
qe20.lycos.com - - [18/Nov/1996:17:20:56 -0600] "GET /robots.txt HTTP/1.0" 200 703
qe20.lycos.com - - [18/Nov/1996:17:21:18 -0600] "GET /robots.txt HTTP/1.0" 200 703
qe20.lycos.com - - [18/Nov/1996:17:26:15 -0600] "GET /robots.txt HTTP/1.0" 200 703
qe20.lycos.com - - [18/Nov/1996:17:26:41 -0600] "GET /robots.txt HTTP/1.0" 200 703
qe20.lycos.com - - [18/Nov/1996:17:28:17 -0600] "GET /robots.txt HTTP/1.0" 200 703
qe20.lycos.com - - [18/Nov/1996:17:32:26 -0600] "GET /robots.txt HTTP/1.0" 200 703
qe20.lycos.com - - [18/Nov/1996:17:32:32 -0600] "GET /robots.txt HTTP/1.0" 200 703
qe20.lycos.com - - [18/Nov/1996:17:32:56 -0600] "GET /robots.txt HTTP/1.0" 200 703
qe20.lycos.com - - [18/Nov/1996:17:33:07 -0600] "GET /robots.txt HTTP/1.0" 200 703
qe20.lycos.com - - [18/Nov/1996:17:33:26 -0600] "GET /robots.txt HTTP/1.0" 200 703
qe20.lycos.com - - [18/Nov/1996:17:38:30 -0600] "GET /robots.txt HTTP/1.0" 200 703
qe20.lycos.com - - [18/Nov/1996:17:39:10 -0600] "GET /robots.txt HTTP/1.0" 200 703
qe20.lycos.com - - [18/Nov/1996:17:39:30 -0600] "GET /robots.txt HTTP/1.0" 200 703
qe20.lycos.com - - [18/Nov/1996:17:39:54 -0600] "GET /robots.txt HTTP/1.0" 200 703
qe20.lycos.com - - [18/Nov/1996:17:43:14 -0600] "GET /robots.txt HTTP/1.0" 200 703
qe20.lycos.com - - [18/Nov/1996:17:43:31 -0600] "GET /robots.txt HTTP/1.0" 200 703
qe20.lycos.com - - [18/Nov/1996:17:49:25 -0600] "GET /robots.txt HTTP/1.0" 200 703
qe20.lycos.com - - [18/Nov/1996:17:56:43 -0600] "GET /robots.txt HTTP/1.0" 200 703
qe20.lycos.com - - [18/Nov/1996:18:06:06 -0600] "GET /robots.txt HTTP/1.0" 200 703
qe20.lycos.com - - [18/Nov/1996:18:07:18 -0600] "GET /robots.txt HTTP/1.0" 200 703
qe20.lycos.com - - [18/Nov/1996:18:08:08 -0600] "GET /robots.txt HTTP/1.0" 200 703
qe20.lycos.com - - [18/Nov/1996:18:14:22 -0600] "GET /robots.txt HTTP/1.0" 200 703
qe20.lycos.com - - [18/Nov/1996:18:43:08 -0600] "GET /robots.txt HTTP/1.0" 200 703
qe20.lycos.com - - [18/Nov/1996:18:53:07 -0600] "GET /robots.txt HTTP/1.0" 200 703

_________________________________________________
This messages was sent by the robots mailing list. To unsubscribe, send mail
to robots-request@webcrawler.com with the word "unsubscribe" in the body.
For more info see http://info.webcrawler.com/mak/projects/robots/robots.html