Re: HOST: header

Scott 'Webster' Wood (swood@thewild.com)
Wed, 17 Jul 1996 08:03:49 -0400 (EDT)

Messages sorted by: [ date ][ thread ][ subject ][ author ]
Next message: Kosta Tombras: "Robot Research"
Previous message: Scott 'Webster' Wood: "Re: Suggestion to help robots and sites coexist a little better"
In reply to: Rob Hartill: "HOST: header"
Next in thread: Rob Hartill: "Re: HOST: header"
Reply: Rob Hartill: "Re: HOST: header"

>
>
> A request from an Apache developer (not me)..
>
> Please could robot owners start sending HOST headers. Aparently this'll
> help server admins to filter out bad URLs. This is in everyone's interest,
> unless there's someone out there that thinks robot writers have a right
> to use whatever URLs they want (nothing would surprise me any more).
>
> thanks.
> rob
>
You may want to tell your friend that they can configure the apache server
itself to restrict access for hosts without the need for robots.txt. Also,
a simple firewall instruction can restrict such requests from specific hosts.

Scott

Next message: Kosta Tombras: "Robot Research"
Previous message: Scott 'Webster' Wood: "Re: Suggestion to help robots and sites coexist a little better"
In reply to: Rob Hartill: "HOST: header"
Next in thread: Rob Hartill: "Re: HOST: header"
Reply: Rob Hartill: "Re: HOST: header"