I've been using my /robots.txt scanner also as a scanner for web
server software data. I've recently optimized it, and discovered in
the process, that the Microsoft Internet Information Server does
*not* send normal info (Time, Content-Length, Server, etc) when it
discovers a "404 Not Found" page.. Is this a bug in MSIIS, or is it
normal that servers don't send this data when reporting errors?
Thanks,
Jakob Faarvang