RE: What to rate limit/lock on, name or IP address?

Greg Fenton (gregf@netcom.ca)
Wed, 23 Oct 1996 14:19:57 -0400


From: Aaron Nabil[SMTP:nabil@teleport.com]
Subject: What to rate limit/lock on, name or IP address?

It's would be undesirable to lock against both the IP address and
the site name. So I won't.

Should I switch from IP address to site name, or leave it as it is?

This is the heart of the site-lock problem for most multi-site crawlers. And
from the study I did, there was just as many problems locking against IP
addresses as against hostnames. There may be multiple IP addresses
per hostname, multiple hostnames per IP address, or both.

I didn't come to a proper conclusion, but I ended up sticking with IP address
locking. The proper thing would be to uniquely identify each "site". This
would involve a lot of work, playing around with forward and reverse name
lookups and a fair amount of by-hand editing.

I don't think that there is a simple way to do it otherwise (suggestions?),
though this is one of the extensions to the REP that I had proposed a while
back.

gregf.