Re: RFC, draft 1

Hrvoje Niksic (hniksic@srce.hr)
16 Nov 1996 20:48:36 +0100

Denis McKeon (dmckeon@swcp.com) wrote:
> Scenario: server admin notices that some agents fetch /robots.txt, but
> fail to honor contents of it - so admin has server respond to request
> for /robots.txt by those agents with the output of a script, which is
> arranged to encourage a poorly written agent to spend a lot of time
> parsing the output. The web server is trying to deny access to its
> services to the agent.

What do you mean by "poorly written agent spending a lot of time
parsing the output"? Is the output simply long, or is it devious in a
way? How can it be, with the current simple specifications?

Can you give an example of such a /robots.txt file?

-- 
Hrvoje Niksic <hniksic@srce.hr> | Student at FER Zagreb, Croatia
--------------------------------+--------------------------------
"What is the sound of Perl?  Is it not the sound of a wall that
people have stopped banging their heads against?" -- Larry Wall
_________________________________________________
This messages was sent by the robots mailing list. To unsubscribe, send mail
to robots-request@webcrawler.com with the word "unsubscribe" in the body.
For more info see http://info.webcrawler.com/mak/projects/robots/robots.html