Re: nastygram from xxx.lanl.gov
Rob Hartill (robh@imdb.com)
Wed, 10 Jul 1996 22:19:53 -0600 (MDT)
>The administrators at xxx.lanl.gov could save themselves an awful lot of
>time by putting the majority of intensive CGI pages behind quasi-forms.
>
> http://xxx.lanl.gov/abs/hep-ph?9606247
>
>instead of the current
>
> http://xxx.lanl.gov/abs/hep-ph/9606247
>
>Bingo -- most robots won't bother indexing past hep-ph, and the
>administrators can spend their valuable time on matters other than exacting
>revenge upon people for inadvertently causing xxx.lanl.gov's lack of skill
>in site design to become obvious.
This covers one class of robot.. the type that ignores robots.txt and
has half a brain cell when it comes to "?" in URLs. Now what about the
other dumb robots (there's a never ending supply) that don't have
this state-of-the-art intelligent ability to look for "?"s
Stop passing the buck. Fix the robots to behave. Monitor their actions.
rob