In addition to endorsing Reinier's pointed remark, I'd like to ad that you
should be aware that your robots will likely find many references to the
same pages, in addition to ".." references. An essential part of any
well-behaved robot is a database of some sort that keeps track of which
pages have been visited, so that you don't waste your time with duplicate
efforts... and more importantly, you don't waste others' bandwidth and
cycles.
This is basic stuff. If you're writing a new robot, you'd do well to push
the envelope a bit by figuring out means of avoiding inclusion of the more
subtle duplicates created by symbolic links, etc., rather than re-inventing
capabilities that are well established.
Nick