Re: User-Agent
Klaus Johannes Rusch (e8726057@student.tuwien.ac.at)
Wed, 20 Nov 1996 23:56:14 CET
In <3292FF88.2906@ozemail.com.au>, David Banes <dbanes@ozemail.com.au> writes:
> Hi,
>
> I'm new to this mailing list, I've been reading for a few days and have
> read Martijn's RFC, now and have a question.
>
> If I'm developing a bot and using OLE Automation (initially with
> Netscape Navigator) and want to conform to robots.txt (SRE?).
>
> Does anyone have any suggestions as to how I can handle the User-Agent
> part. You'll all be aware that Navigator is going to announce itself as
> Mozilla. The only way round this that I can see is to use 'normal'
> sockets code to send my own User-Agent header down to the server BEFORE
> firing off Navigator. At least I can then get the bot 'registered' with
> each server I hit, just in case someone wants to complain :)
You could set up Netscape to go via a proxy server, and then create a minimal
proxy server stripping off the User-Agent header and inserting it's own one,
or at least say something like
"Mozilla/... via proxy TheOLERobot/0.01"
Modifying a server to act as such a proxy should be fairly simple (even with
fully replacing the header line).
Klaus Johannes Rusch
--
e8726057@student.tuwien.ac.at, KlausRusch@atmedia.net
http://www.atmedia.net/KlausRusch/
_________________________________________________
This messages was sent by the robots mailing list. To unsubscribe, send mail
to robots-request@webcrawler.com with the word "unsubscribe" in the body.
For more info see http://info.webcrawler.com/mak/projects/robots/robots.html