|
Ethan,
Thanks for the response. I'm hoping to give the catalog generation
another try, so I will look into your suggestions. Our fallback is just to
write some scripts to generate the xml for our catalog manually...doesn't look
too bad, except that it wouldn't stay current with any changes you make for the
schema I suppose. We'll figure something out. Still looking at
catalog definitions for OGC services as well.
thanks,
Ken
-----Original Message----- Et
From: Ethan Davis
[mailto:address@hidden]
Sent: Fri 02/07/2003 02:23 PM
To: Ken Keiser
Cc: Robert Casey;
address@hidden
Subject: Re: THREDDS
update
Hi Ken, all,
I think you tried the generator back
when it was using XSLT. The current
version is using the catalog library
that John wrote which uses JDOM. I haven't
tested it on large collections
like yours. But since JDOM creates an in-memory
document, I expect there
will be similar memory issues. However, I suspect that
the JDOM version
will be a bit less memory intensive than the XSLT version.
Though probably
not enough to help with your large data collection.
I have two possible
solutions to suggest for these situations. First, the
catalog could be
split into smaller catalogs by using catalogRef elements. For
instance, you
could have a top level catalog that showed the hierarchical
structure of
your data collection but all the leaf nodes were catalogRefs
pointing to
individual catalogs for each lower level collection. However, the
generator
doesn't yet support yet the use of catalog refs. So, you would have to
run
the generator for each catalogRef directory.
Second, we are working on
Data Query Capability (DQC) documents for the case
were a single level of
the hierarchy contains large numbers of datasets;
catalogRefs just won't
help. Besides the memory issue with the generator, a
parser can get bogged
down and, more importantly, a user will struggle to sort
through a huge
list of datasets. We're seeing this with our radar data which has
several
products coming in every second. Basically, the DQC allows us to
describe a
set of allowed queries. For instance, with our radar dataset we can
let the
user ask for the last hour of base reflectivity data from one
specific
station. A catalog listing just those datasets that satisfy the
request is
returned.
There's more on our DQC work on the THREDDS
tech page:
http://www.unidata.ucar.edu/projects/THREDDS/tech/#DQC
Sorry
to be so long winded. Hope this has been useful
information.
Ethan
Ken Keiser wrote:
>
> At the time
it was memory errors, but as I said that was several months
> ago and
I'm sure there are newer versions we have not tried since.
>
>
-----Original Message-----
> From: Robert Casey [mailto:address@hidden]
>
Sent: Tuesday, February 04, 2003 9:58 AM
> To: Ken Keiser
> Cc:
Ben Domenico; address@hidden
> Subject: RE: THREDDS
update
>
> > number of years. The generator choked on the
large number of
> > directories and granules. We will download
the latest version and
try
>
> What kind
of error did you get? Array out of bounds? Out of
>
Memory
> error?
>
>
------------------------------------------------------------------------
>
Robert Casey, Software
Engineer web:
www.iris.washington.edu
> IRIS Data Management
Center
email: address@hidden
> 1408 NE 45th
St.
phone: (206) 547-0393 x117
> Seattle, WA
98105
fax: (206) 547-1093
>
------------------------------------------------------------------------
--
Ethan
R.
Davis
Telephone: (303) 497-8155
Software
Engineer
Fax: (303) 497-8690
UCAR Unidata
Program Center
E-mail: address@hidden
P.O. Box 3000
Boulder,
CO
80307-3000
http://www.unidata.ucar.edu/
|