So you don't have to notify all robots it has to be local - maybe a file at
root level "updated.txt" which does something like the AFTP templates..
If it's absent you update the lot (and maybe make an assumption that the
server is not a serious player - whatever that means in terms of
consequences).
If it's there the site gains though less traffic and the robot gains also.
Spammers do what spammers do and you can't prevent it.
Tony
______________________________________________
Email : Tony.Barry@anu.edu.au
Voice : +61 6 249 4632 Fax: +61 6 279 8120
Details : http://snazzy.anu.edu.au/People/TonyB.html
Head, Center for Networked Access to Scholarly Information,
Australian National University Library, A.C.T. 0200, AUSTRALIA.
_________________________________________________
This messages was sent by the robots mailing list. To unsubscribe, send mail
to robots-request@webcrawler.com with the word "unsubscribe" in the body.
For more info see http://info.webcrawler.com/mak/projects/robots/robots.html