> > -[someone else wrote:]
> > I was also thinking that something we all might collaborate on would be a
> > list of sites that as a robot you shouldn't index, or should only index the
> > top levels. This might save the net a lot of bandwidth, and everyone a lot
> > of hassle. We could develop a database to hold this information. Besides
> > being useful to us, it might provide a good forum to show site designers
> > why they should or shouldn't do certain things with real examples.
>
> The above sounds like a good idea. Volunteers to set it up?
>
I think there will be a problem keeping up with all these kinds of
sites. Web systems with dynamic data in the path-info are easy and fun
for the programmers. I think there will be TONS of these sites very soon.
> I'd also be interested in making a list of sites that you could index as part
> of the testing process. With more and more robot developers, it could be a
> great resource to have a page of URLsof people with servers they don't mind
> getting hit a lot with robot traffic, or which have special traps or other
> devices setup on them for testing purposes.
>
> Issac
>
I would be open to some testing on my site, which has record keys and
cache-defeating counters in the path info. Any robot that waits-I don't
know, I'm not a webmaster- a half minute or so between accesses, is
welcome to ignore the robots.txt files around my site for the purposes
of testing, if you don't index the disallowed stuff publicly.
-Ann Cantelow
cantelow@athena.csdco.com
-------------------The Interactive Poetry Pages----------------------
Collaborative poetry in real time- across the net.
http://www.csd.net/~cantelow/poem_welcome.html
---------------------------------------------------------------------