>THe question is -- how to make this information available to
>an indexing engine?
just for completeness, you may want to check out InfoSeek's siteinfo.txt (?)
mentioned in a recent posting. And for historical interest there is always
ALIWEB :-)
-- Martijn
Email: m.koster@webcrawler.com
WWW: http://info.webcrawler.com/mak/mak.html
_________________________________________________
This messages was sent by the robots mailing list. To unsubscribe, send mail
to robots-request@webcrawler.com with the word "unsubscribe" in the body.
For more info see http://info.webcrawler.com/mak/projects/robots/robots.html