Re: Servers vs Agents

David M Banes (dbanes@ozemail.com.au)
Fri, 29 Nov 1996 10:12:12 +1100

Erik Selberg wrote:
>
> Martin Kiff <mgk@webfeet.co.uk> writes:
>
> > (My 'hipcrime' analogy is that it is equivalent to a driver who does not
> .....
> > because you don't want it done to you)
>
> My analogy is that the robots.txt standard is like USENET
> FAQs. They're great for people who know about them and abide by
> netiquette to look up answers there first, but for the throngs of
> people out there, they still all ask the same questions over and over
> and over and over....
>
> The simple truth is that you _need_ some method of enforcement of your
> robots.txt for people who are ignorant. You need to ensure folks are
> aware of it, and you need a strong incentive for people to use
> it. Right now, the only real incentives out there are that if you
> don't follow it, Paul and Rob will send you hate mail after the
> fact. If the robot author is able to obtain what he wants, then it
> seems perfectly reasonable to me that folks will choose not to support
> the standard (assuming they even care to look to see if there is one).
>
> -Erik

I'm writing a user agent, via Netscapes OLE Automation interface, I have
a dilemma. I want to abide by some standard (I'll use the original one
until something else comes along) but at the same time I don't want my
customers to feel they are using a 'crippled' product because it can't
get at stuff they can manually.

BUT, as a spider/crawler/agent author and someone who care's about
other's I'm going to have to support robots.txt, so user education, i.e.
information supplied with the software explaining why they shouldn't
leave it running overnight with no delays between hit's, is one way I'm
going to address the contradiction of a user 'need' for information/data
now and NOT trampling all over peoples sites with rapid-fire requests.

I'd like to think I'm representative of the majority of developers in
this field.

I need help getting my User Agent: header to a server via the Navigator
interface, all attempts have failed to date, I'm afraid I may be
releasing the software and you'll only know you've been hit if you see
lots of Mozilla/??? hits in sequence.

Any help would be appreciated... of course I could put all the old
sockets stuff in, but that seems a waste when I'm using Netscape's OLE
Automation.

David.

-- 
[--------------------------------------------------------------------]
 email:dbanes@ozemail.com.au       CServe - 100446.102@compuserve.com 
 http://www.ozemail.com.au/~dbanes MSN    - banes@msn.com
[--------------------------------------------------------------------]


_________________________________________________
This messages was sent by the robots mailing list. To unsubscribe, send mail
to robots-request@webcrawler.com with the word "unsubscribe" in the body.
For more info see http://info.webcrawler.com/mak/projects/robots/robots.html