Re: How frequently should I check /robots.txt?
Martijn Koster (mak@surfski.webcrawler.com)
Mon, 27 Nov 1995 23:22:54 -0800
Deciding how often to check for the /robots.txt depends highly on how
you run your robot: how many runs per week, how many documents when,
etc. I'd say a week is a reasoneable time. If your robot supports
end-user submissions you could of course be clever about people
submitting their /robots.txt URL; that would give them more influence.
-- Martijn
__________
Email: m.koster@webcrawler.com
WWW: http://info.webcrawler.com/mak/mak.html