I have a script that would like to show one thing to a person and
something slightly different to a web crawler. Is there some way for
it to know?
At http://info.webcrawler.com/mak/projects/robots/guidelines.html it says:
Identify yourself
HTTP supports a From field to identify the user who runs the WWW
browser. Use this to advertise your email address
e.g. "j.smith@somehwere.edu". This will allow server maintainers to
contact you in case of problems, so that you can start a dialogue on
better terms than if you were hard to track down
I checked that my CGI gets an HTTP_FROM environment variable if I pass
one by doing the HTTP query by hand, and that it does not get
nonempty HTTP_FROM envariables from queries coming from Netscape 2.02,
Mosaic 2.6, and Lynx 2.4.2. I checked the pages near the
guidelines.html URL above to check that the common webcrawlers pass
from fields like they should.
Are there any browsers out there that pass something for HTTP_FROM?
Is there a better idea than assuming that the accessor is a web
crawler if there is an HTTP_FROM field?
If not, then I'm all set.
Thanks,
Tim Freeman