> I've read in the FAQ that one should only ask for what one can
> handle, using the "Accept:" field.
Does a crawler really benefit from using the Accept: field? I have yet to see a
link which would actually work better if fetched with an Accept: text/html or
something.
-Ed-
mred@neosoft.com
http://www.neosoft.com/~mred/