Request for Source code in C for Robots

ACHAKS (ACHAKS@.LOCAL.Inf.COM)
Tue, 26 Mar 96 18:13:21 EST


Hi friends,

I am planning to make a robot(Window3.1 based) using the NetAPI calls.
I could extract the url but I need to parse it to get subsequent
url's.
Is there any public domain(shareware or freeware) parser(in C) available for
the same which I can use?
Is there any source code for robots in C available which I can go through ?
It would be better if it implements robots exclusion protocol.

Thanks in anticipation,

Angs.