Re: robots.txt syntax

Martijn Koster (m.koster@webcrawler.com)
Fri, 11 Oct 1996 09:18:54 -0700

Messages sorted by: [ date ][ thread ][ subject ][ author ]
Next message: Aaron Nabil: "Another rating scam! (And a proposal on how to fix it)"
Previous message: John D. Pritchard: "Re: robots.txt syntax"
Maybe in reply to: Fred K. Lenherr: "robots.txt syntax"
Next in thread: Martijn Koster: "Re: robots.txt syntax"

At 9:53 AM 10/11/96, John D. Pritchard wrote:
>> Disallow: /
>> Allow: /public/
>
>a couple of people, including myself, have proposed this. one of us needs
>to get around to writing a draft extending the present draft on
>robots.txt.

Funny you should say that -- I'm adding support for that to my robots.txt
library + test-suite tomorrow.

I'll see how far I get with writing it up to; I really don't want to
come out with a new spec without fixing some other things.

I now have a FreeBSD/Win95 box at home -- that helps :-)

-- Martijn

Email: m.koster@webcrawler.com
WWW: http://info.webcrawler.com/mak/mak.html

Next message: Aaron Nabil: "Another rating scam! (And a proposal on how to fix it)"
Previous message: John D. Pritchard: "Re: robots.txt syntax"
Maybe in reply to: Fred K. Lenherr: "robots.txt syntax"
Next in thread: Martijn Koster: "Re: robots.txt syntax"