> Web Server:
> 1) New document is submitted.
>
> 2) Web server adds document to SFT.txt, RSA.txt, and a third un-named
> file containing the Document Meta Information (lets call it DMI).
nearly .. this version create a new meta file for each file submitted
the name of this file is :".<filename>.meta" but isn't important because
it's the server that handle this file to send the meta-info in the HEAD
part of the HTTP HEAD answer
> 3) When HEAD request comes in, pulls meta info from the DMI file, adds in
> the regular headers and sends it out.
> [...]
> It seems that if you are already defining two files, you could just as
> easily define three, and add a DMI.txt that contained all meta
> information for each document on the system.
> What you would gain.
> 1) With the request of three documents, a robot could index your entire site
> 2) Lower server load. You wouldn't have to reparse the DMI to generate
> information for each HEAD request, and there would be many fewer requests
> from robots.
> What you would lose.
> 1) the robot probably wouldn't get the last-modified date and content length
> [...]
I haven't used a dmi.txt file for principally for one reason:
If I have to parse each time a file (it could be really big!) to
extract the meta info, I'll have some problems with the performance of the
server.
I think that a managment of a DMI file could represent a big overload
for a server also if a robot could benefit from this.
A solution could be that an automatic process build the DMI file every
night (?) using all the ".<filename>.meta" files.
But doing so we have increased by a factor 3 the redundace of the meta
information (we can find their in the HTML files, in the
".<filename>.meta" files and in the DMI.txt file).
We can gzip this file, but the problem of the redundance remain.
Davide
-----------------------------------------------------------------------------
Davide Musella
Institute for Multimedia Technologies, National Research Council, Milan, ITALY
tel. +39.(0)2.70643271
e-mail: davide@jargo.itim.mi.cnr.it http://jargo.itim.mi.cnr.it/