205.252.60.5[0-8]
who call themselves 'Merritt/1.0', don't look at /robots.txt
and don't have 'referrer' details. (205.252.60.71 is mentioned in
the 'List of Robots' but as an address which does read /robots.txt.
Same subnet, hmmmm).
Is there any way of tracing these addresses back to source as
they are scanning areas where pages used to be and are no longer -
and picking up a "302 redirect".
If I know they are useful robots (and likely to make use of the
302 redirect) I'll carry on as I am. If they are useful and are going
to ignore the 302 I'll feed back a "404 Page departed" or something.
Is there any way of finding this sort of information out? How do
robots (generally) handle 302's? and all sorts of other questions :-)
Regards,
Martin Kiff
mgk@newton.npl.co.uk
P.S. Sorry to pop up in the list and start a quick-fire list of questions,
point me at references if this has been covered recently. Finding the
<Meta ...> discussion/dissection highly useful, thanks....