I'm new to this mailing list, I've been reading for a few days and have
read Martijn's RFC, now and have a question.
If I'm developing a bot and using OLE Automation (initially with
Netscape Navigator) and want to conform to robots.txt (SRE?).
Does anyone have any suggestions as to how I can handle the User-Agent
part. You'll all be aware that Navigator is going to announce itself as
Mozilla. The only way round this that I can see is to use 'normal'
sockets code to send my own User-Agent header down to the server BEFORE
firing off Navigator. At least I can then get the bot 'registered' with
each server I hit, just in case someone wants to complain :)
Regards
David Banes
[--------------------------------------------------------------------]
dbanes@coolbananas.com.au http://www.ozemail.com.au/~dbanes
[--------------------------------------------------------------------]
_________________________________________________
This messages was sent by the robots mailing list. To unsubscribe, send mail
to robots-request@webcrawler.com with the word "unsubscribe" in the body.
For more info see http://info.webcrawler.com/mak/projects/robots/robots.html