Re: Does this count as a robot?

Jeremy.Ellman (jeremy@mari.co.uk)
Mon, 8 Jan 1996 09:59:44 GMT

Messages sorted by: [ date ][ thread ][ subject ][ author ]
Next message: Benjamin Franz: "Re: Does this count as a robot?"
Previous message: Thomas Stets: "Does this count as a robot?"
Maybe in reply to: Thomas Stets: "Does this count as a robot?"
Next in thread: Benjamin Franz: "Re: Does this count as a robot?"

>
> I am currently writing (actually it's already written and I'm testing)
> a program to copy a subtree of a server to my machine.
>

Sounds like HTMLGOBBLE. Why not just do use that? I've been trying to fix some bugs in it
but it's only real problem is that it does not respect robots.txt

Jeremy Ellman

MARI Computer Systems

Next message: Benjamin Franz: "Re: Does this count as a robot?"
Previous message: Thomas Stets: "Does this count as a robot?"
Maybe in reply to: Thomas Stets: "Does this count as a robot?"
Next in thread: Benjamin Franz: "Re: Does this count as a robot?"