Web pages being served from an SQL database

Eric Mackie (eric@watfac.org)
Tue, 17 Dec 1996 15:04:32 -0500

I have a web site (www.livepage.com) which is serving up pages from an =
SQL database rather than directly from the file system. All of the URLs =
in the site have the syntax of a CGI program (although I am really using =
NSAPI so this syntax was just for convenience). In all other ways it is =
a well-behaved web site.

My problem is that none of the major web search robots will seem to =
index the site except for the home page. I assume that this is because =
the URLs look like CGI programs. Is this the case? I haven't been able =
to find any information on what URLs are typically ignored other than =
that Lycos mentions that it doesn't look at any URLs that contain "?". =
Is there any way to convince the robots that it is OK to go ahead and =
traverse these URLs? Does anyone have any suggestions as to what would =
be the right solution?

It seems to me that this is a problem that will come up more and more =
often as people start using more complex servers. Has anyone figured out =
a solution?

Thanks
Eric Mackie
_________________________________________________
This messages was sent by the robots mailing list. To unsubscribe, send mail
to robots-request@webcrawler.com with the word "unsubscribe" in the body.
For more info see http://info.webcrawler.com/mak/projects/robots/robots.html