> >what are the implications of the general public having a Webcrawler?
[snip]
>
> lets not look at what will happen when everyone will have one that
> is the same question people ask about every product, what will happen
> when everyone can own a car, oh no, the railroads will go out
> of business, what will happen when someone makes the first
> personal computer, oh no mainframes become obsolete.
You have missed the point. It is not about the economic impact of personal
webcrawlers on the big indexes. It is about the problem of umpteen
*MILLION* robots all attempting to access every site on the web. The
small number of robots today make a noticible (albeit small) dent in
overall system loading of websites. One that is more than offset by their
usefulness to sites as resource discovery. But there are only a double
handful of them running. If as few as two hundred comprehensive and fast
independant webcrawling robots like Scout were set loose the traffic they
generated on would swamp everything else percentage wise. This would make
them too much of a burden on the websites they index - sites would begin
implementing measures to block all robots except the big known ones. A
system such as Harvest where duplication of effort is minimumized though
sharing of gathered information is required to solve this problem.
-- Benjamin Franz