>
> > What do you mean by "Someone published the URL"? Do you mean that someone
> > has to explictly link to these pages, or that these files are simply located
> > in the data directory of the Web Server?
>
> Unfortunately, same thing. If someone publishes the directory, AND the
> directory browsing feature on the server is not disabled, THEN scooter will get
> a nicely formatted html page containing pointers to every file in the directory,
> and there is no easy way for me (short of really ugly heuristics) to detect the
> situation. But I wish I could. Plea to most webmasters: please please please,
> disable the directory indexing feature, it is rarely needed, and causes robots
> to pick a lot of junk.
>
You can try to scan for specific string such as
"up to previous dirs " or similar one