I think that all the operations of a "robot",
(even HEAD requests ) need to conform in "robot guidelines" (robots.txt, etc),
since these requests need "attacked" system resources.
459 HEAD requests produce 459 server child-forks. Many of the requests
will operate in papallel. If the server has an upper limit in forking maybe
the site will be saved, else the paging volume will be full.
In our site we had a similar problem with HotBot, even with an AIX with 64M
of memory and a lot of free disk (and upper limit in server).
My opinion is that:
a "robot" need to conform in any case with robots.txt and
request from a site a subset of URLs (in a logical rate), not all of them at once.
>----------
>From: Aaron Nabil[SMTP:nabil@i.net]
>Subject: nastygram from xxx.lanl.gov
>
>I just got a nastygram from the web admin at xxx.lanl.gov accusing
>my "robot" of "attacking" him. This "attack" consisted of HEAD's on
>459 URL's, with a mean pause of about 2 minutes. The total data set
>was (all sites) 653k URLs, and yes I probably should have filtered the test set to
>limit the number of accesses to any one site. Mea culpa.
>
>He accused my "robot" of violating the "robot guidelines". He didn't
>enumerate which I violated. I'm guessing he may have been upset that the
>test was ignoring his robot.txt, but since the test wasn't traversing the
>general web space and was in no danger of looping or getting lost, there wasn't
>much point. The test was operating from a fixed list of URLs that once tested
>would be discarded.
>
>He also informed me that "we have no need for you to 'index' our site" only
>to then rebuke me for running "a particularly stupid robots that only does
>pointless HEADs". I didn't point out that the two would naturally be mutually
>exclusive. :)
>
>Anyway, he naturally asked that I "cease and desist", which I'm of course I'm
>happy to do.
>
>Comments? Should such accesses as mine also test robots.txt? Were my accesses
>"burdensome" at that rate? Are the "robot guidelines" no longer "guidelines"
>but "rules" and are these rules applicable to all forms of automated access, even if
>they aren't robots?
>--
>nabil@i.net
>http://www.i.net/aaron/
>
------ =_NextPart_000_01BB6D8E.D0C0CAE0
Content-Type: application/ms-tnef
Content-Transfer-Encoding: base64
eJ8+IisJAQaQCAAEAAAAAAABAAEAAQeQBgAIAAAA5QQAAAAAAADpAAENgAQAAgAAAAIAAgABBJAG
ACQBAAABAAAADAAAAAMAADADAAAACwAPDgAAAAACAf8PAQAAAEkAAAAAAAAAgSsfpL6jEBmdbgDd
AQ9UAgAAAAByb2JvdHNAd2ViY3Jhd2xlci5jb20AU01UUAByb2JvdHNAd2ViY3Jhd2xlci5jb20A
AAAAHgACMAEAAAAFAAAAU01UUAAAAAAeAAMwAQAAABYAAAByb2JvdHNAd2ViY3Jhd2xlci5jb20A
AAADABUMAQAAAAMA/g8GAAAAHgABMAEAAAAYAAAAJ3JvYm90c0B3ZWJjcmF3bGVyLmNvbScAAgEL
MAEAAAAbAAAAU01UUDpST0JPVFNAV0VCQ1JBV0xFUi5DT00AAAMAADkAAAAACwBAOgEAAAACAfYP
AQAAAAQAAAAAAAAD0jcBCIAHABgAAABJUE0uTWljcm9zb2Z0IE1haWwuTm90ZQAxCAEEgAEAIAAA
AFJFOiBuYXN0eWdyYW0gZnJvbSB4eHgubGFubC5nb3YAcgsBBYADAA4AAADMBwcACQAMAAQAKQAC
AB4BASCAAwAOAAAAzAcHAAkACwAuAAgAAgAmAQEJgAEAIQAAADZFRTU1QTcyN0VEOUNGMTFBNDEz
NDQ0NTUzNTQwMDAwAPMGAQOQBgAkCQAAEgAAAAsAIwAAAAAAAwAmAAAAAAALACkAAAAAAAMANgAA
AAAAQAA5AKDCV6p1bbsBHgBwAAEAAAAgAAAAUkU6IG5hc3R5Z3JhbSBmcm9tIHh4eC5sYW5sLmdv
dgACAXEAAQAAABYAAAABu211qk5yWuVv2X4Rz6QTREVTVAAAAAAeAB4MAQAAAAUAAABTTVRQAAAA
AB4AHwwBAAAAGgAAAGVsc2lkQGFyY2hpbWVkZXMueS1uZXQuZ3IAAAADAAYQMy4gJAMABxC9BgAA
HgAIEAEAAABlAAAASVRISU5LVEhBVEFMTFRIRU9QRVJBVElPTlNPRkEiUk9CT1QiLChFVkVOSEVB
RFJFUVVFU1RTKU5FRURUT0NPTkZPUk1JTiJST0JPVEdVSURFTElORVMiKFJPQk9UU1RYVCxFVAAA
AAACAQkQAQAAAJoHAACWBwAA+xEAAExaRnX3wQ+r/wAKAQ8CFQKoBesCgwBQAvIJAgBjaArAc2V0
MvUAACoC4WEHgAYABsMCgGxNVBKHAfEyA8UCAHAIcnEyEoFzdGVt3QKDMwLkBxMCgH0KgAjPxQnZ
OxdPMjU1AoAKgYMNsQtgbmcxMDMVkNcLChWRC/JjAEAgCoUcklBJIHRoC4BrHPFhnwVAB0ADIB0A
EnBvcASQTR2AaQIgBCBvZh2gIEYiA2AG4HQiLBw5KARldgnwIEhFQUQ6IBdQcQpQFQAEICkgSm4J
4GQc8G8gBaBuMQIQcm0gC4AfJSBntHVpDbBsC4AHkCIgUIkfQ3MuDNB0LCARwMxjKR+qAJBuYxJw
HfE3EbAhCCHDIh2AAZBja9sJgCQAcxTzIQFzCGEmIMckgAqFHDk0NTkgvRSQ3QRwdSYhKnIRsHIg
gAXA0xFwAxBkLSJxaySABdD9AHB5HtId8iEXHDkD8B3B/x40EnAiwQqwCrAdwCOgLXA+SS30LFUR
gAQgA5F1cOUeQSAjsG1pBUAiwS0iyQuAZyAAwHliEnAcOZ8xYzMQEnAvkzQhc2EggLpkJNFsJoEd
8gqwZzOymnYG8HUSYTWWZnUdwP8pTxzBA6AIYTU1EnARgCHw/x8QAJAzAAtgBcArkQJgKIHvA/Ad
ACCwH3BCH3Ak0SCCIzwzA5FBSVg8JDY03k0cKh7hB4AEYHItwABwPzryFuAFQB7hA1AJ4CBkuwQA
HUAoP/IyjSxUKTi/9RyQTS3BcAuAHpEisAQgfR1iOhw5HwghvyLBLaJj/zIgNXI8USQ4P+JFeiEV
QLHPA3A7AzAxOxF1YhGxHtLoVVJMBCAoR9JAQTcw/0hAAyAwEiUgRsAjMR2yLeT/SzFAcSYRQx1Q
HymJCvQjsAgxODAC0WktMTSeNA3wDNBTQxsfID4LSlwxNgqgA2AVEGMFQC1/VqcKh1QeVhdVLgww
ViZG/QNhOlenVb0MghXwCsBEwSZOAaADEFtTExBQOiMSQF2hQGkuRtB0Xa9XT1hfWW9WF1NL8GpW
Ye9bT1xbEkAVAHkJwBJQSvQaeGXwLhoxMQBnb3afXv9gD1UvVjVR3TM2U+/PaC8bWFYmHOBqdUrR
ZoD/HZFk/h3yOqBdADrgMwBH4ecFQGX6HaBjY27QM7Fmv/dsH20vVjVtLcBGVh7hJ5WXM7EkAB0Q
bS1wIFQdEP8EICeVJAAiQQCQFQEh8B7h9SDCJx7BbnLPc99071Y1/ypyTIF6UCTQPVQ/cQORCrDv
btAeER7xBuB1BUAUwHEBv4BwKSF4QiYxH3BNgWQdgL87ERHAeq97v3zPVjV3MiEXQWAdwTVCcyGg
NjUz7x1ATIIk0D/yeQeRHOA7suUBoGwtwHNoCGAs8DIB/yCAM2ADEBUQF1Ed44EBBUD/TBIiEIKP
g5+Er1Y1MuQd8v+A4AbQLJEe4nJQB5ARsEUB3yIgLaNG0DUzeDFNf3AiMP84gAqwQxaLv4zPjd+S
f5OPe5SfVjVIS6FyUkbxdmx2vx6QC2AegDPBHfIi/yJ4MeOZsUEQZG4ngn+XH5gv71YmCfA3sTAT
dx0QEXAc0eebRAmAeDFJJyKgI2CQUf8zsh4BM/GJZDQgIJEygIrj/x1yHfGeD58foC9WNYqThiL8
aWdOIAUQpBJE8R9DJJT+YoBxJfepx53RHPAeYCyB36Pzpg+nH6gvbghnCfAeUfsDIHCicwqwJiE/
8qojA6D/TiCB8RpQj8QW4ERyM9AFsf+yAAJAM7IW4BUAJNAd8RdQf60Frm+vf7CPdfYr0DxgcP5v
C4CrcIFFqeYeNTPBSwX9UxB4RvEjsErRTFYdY09S/4qDCYC2/7gPuR+FtokjNCH/QRFIQAsgoyG/
f8CPwZ+Vr//E78X/mN8HQCjQIrEicpoy+yYyHYEiOqOJkbPRRtMicTeIAAhgIgInC4ANsHgn9zoH
dvFmUHnH78j/yg+pZv8iIB3xA6AXUKvgJ/DMgs4i+HJ1bgMAM8EnkDCBACDvTWA4gArAiNJ0MoAj
gEjV875miNFkbweQ0D/RT9Jf/1Y1uuMw4AQRIMKdIxzgnaT/utQ6AaWYHPDDEMMFEkDWwP+yQYjR
NCG6kNbA3/LYf9mP89qfoTZ4YwpAAJAggC1w/Dop4P/iD+Mfx3/mD+cf7W3MQS2whiB5JNCPUt+3
7zIgJ/EdVBzgIiYgSFI/8v8NsHlyH5CiNqOBHuEFoAhw/yaBo3HpD+of6y9WNRGAMpD/LcAiEdgw
xD/yf/OP6I/2bzf3f229CFBtB4ACMHM//wYAiRRL4KJxkCcyIXEBy5XnipMkOP2QIFe2UnZh/nfv
+X/6j/ufVjUiq+DD8Xlg/wgQz8Gl0x2BMBIAwRYANrTfnC9GsSIgFuC0IyIjaQIP/wMfBC9WNavi
HzA4gCPiP/P/B3Qmgg5yHaAykCOwTXA74f+Qox3BInIexIBwBiCjApAV/TzlaR7wCn8LjwyfqWYe
AO8/0RiArUIkND8TbxR/FY//VjcYPxlPGl9WNV45G/8dDwceH/SmtXBwOi8vd6UkUC5eky9hXTIv
IE+/IV8ib2nsal9rbih9fSVQAgAsoAAAAwAQEAAAAAADABEQAQAAAEAABzCA6e8Sc227AUAACDCA
6e8Sc227AR4APQABAAAABQAAAFJFOiAAAAAAXzk=
------ =_NextPart_000_01BB6D8E.D0C0CAE0--