robots.txt

Tangy Verdell (TVerdell@dca.com)
Wed, 14 Feb 96 11:33:33 EST


how standardized is the robots.txt file. has anyone ran into problems where a
significant number of sites have made typos or erros in their robots.txt file.

for example: spelling User-agent as useragent.

______________________________ Reply Separator _________________________________
Subject: The Robot And Search Engine FAQ
Author: robots@webcrawler.com at Internet
Date: 2/13/96 3:03 PM

Due to the size of the file and number of links, I've decided to
just post the url. If anyone would like a copy just yell, I'll mail it.
The Robot and Search Engine FAQ can be found at
http://science.smsu.edu/robot/faq/robot.html

p.s. It's somewhat pollished but, you may still need your construction hat.
Please comment, my goal is to provide the best possible FAQ.

Keith D. Fischer
kfischer@mail.win.org
kfischer@sy.smsu.edu