We have a C (or C++) implementation of a robots.txt parser,
incorporated in our version 2.0 of the Fish-Search, which should be
out in another week or so.
Fish-Search is a cgi-script that performs on-line search.
It is entirely written in C and C++.
It uses the W3C reference library (currently 4.0C), does parallel
retrieval, obeys the robot exclusion protocol, and makes sure not to
bog single servers (unless the whole search is restricted to a single server).
I will make an announcement when the distribution is made public.
Paul.