Similar ideas have been discussed before, I think robots.txt and sitelist.txt
have different goals and can coexist.
robots.txt is for robots who want to index pages themselves, following their
design principles, and need guidance where not to go. Not much potential for
abuse in robots.txt.
sitelist.txt is for robots who trust the owner of a site to compile relevant
summary information. Plenty of potential for abuse in sitelist.txt (similar to
adding unrelated keywords all over the pages).
Klaus Johannes Rusch
-- e8726057@student.tuwien.ac.at, KlausRusch@atmedia.net http://www.atmedia.net/KlausRusch/ _________________________________________________ This messages was sent by the robots mailing list. To unsubscribe, send mail to robots-request@webcrawler.com with the word "unsubscribe" in the body. For more info see http://info.webcrawler.com/mak/projects/robots/robots.html