Servers vs Agents

Davis, Ian (iand@wcom.co.uk)
Wed, 27 Nov 96 15:21:00 GMT


I have been following this list for a while now and recent discussions
have prompted me to add my small voice to the fray. As I see it there is
no way to force robots/agents to use robots.txt. We are on the brink of a
Cambrian explosion of intelligent agents and it is futile to expect every
agent developer to implement support for robots.txt. In addition to the
arrogant few who choose to ignore robots.txt out of choice, there will be
countless more who will not support it because they are unaware of its
existence, its importance or its function.

What I am proposing is that we re-evaluate the reasoning behind
robots.txt. The proposals I have seen in this list seem to rely on the
assumption that robots.txt is enforcible when it is quite clearly not.

One of the primary reasons to restrict access to a site is to prevent
abuse of cpu cycles and/or bandwidth by unscrupulous robots and agents.
There is no way that robots.txt can ever hope to achieve this, certainly
not while clients can choose to ignore it. What is needed are more
intelligent servers to counter the intelligent agents of the future. The
servers should be able to anticipate the load based on experience and
aquired knowledge and dynamically adjust their delivery rates so as not
to overload the network. These servers are going to have to learn how to
prioritise different types of clients based on previous access patterns.
There is a strong evolutionary pressure at work here and the only way to
survive is to adapt.

Ian Davis
iand@fdc.co.uk
--
Figuredata Computing Ltd. | email: info@fdc.co.uk
Making Science Compute | Tel: +44 (0)1932 350554

_________________________________________________
This messages was sent by the robots mailing list. To unsubscribe, send mail
to robots-request@webcrawler.com with the word "unsubscribe" in the body.
For more info see http://info.webcrawler.com/mak/projects/robots/robots.html