20010130: defining ADDE datasets in McIDAS-X

>From: address@hidden (Chris Hennon)
>Organization: Ohio State
>Keywords: 200101301841.f0UIfKX26078 McIDAS ADDE

Chris,

>Hello.  I am a new mcidas user.

This is starting out like an AA introduction :-)

>I've been going through the learning guide and looking at other
>documentation in a vain attempt to understand the structure of how data is
>stored and accessed in mcidas.  Perhaps you can quickly clear up some
>confusion I'm having. 

There are three ways of creating ADDE datasets in McIDAS.  One way
works with data files in McIDAS format and using McIDAS naming
conventions; the second uses data files in McIDAS format that are not
named using McIDAS naming conventions; the third is specific to servers
written to handle data that is not in McIDAS format (and so the names
are not relevant to McIDAS either).

>I have a directory of many area files in raw format that I would like to 
>remap:
>
>twister:[/usr/local/ldm/data/archive/chennon/233-243_1998]% ls
>goes8_1_1998_233_0015   goes8_2_1998_236_1215   goes8_2_1998_240_0615
>goes8_2_1998_233_0015   goes8_2_1998_236_1815   goes8_2_1998_240_1215
>goes8_2_1998_233_0615   goes8_2_1998_237_0015   goes8_2_1998_240_1815
>
>etc...

OK, what you want to do fits the second type above.

>They are located on the same server I'm working on, and right now I don't
>have the need to be able to access them from another machine.

OK.

Since this response will be accessible to all who care to review entries
in the Unidata McIDAS-X online tracking system, I hope that you won't mind
if I am overly verbose about the dataset creation process.

>How would I use DATALOC, DSSERVE, etc.. to access my files?

For the sake of claity:

o ADDE datasets are collections of data of any/all types supported by
  McIDAS: IMAGE, POINT, GRID, and TEXT

o ADDE dataset identifiers are pairs of names: the first part of the pair
  is the 'group'.  A group is composed of subsets of data organized
  by type and referenced by 'descriptor's.  A complete ADDE dataset
  reference is then:

  group/descriptor

In the case you are proposing, the dataset will be composed only of
elements of type IMAGE.  What you did not say is whether or not there
is any logical grouping of data that should be called out by assignment
of multiple 'descriptor's.  For instance, if you have imagery from both
GOES-8 and GOES-10, you might want to organize your dataset into two
subsets, one for GOES-8 and one for GOES-10.

You may want to organize the data by satellite wavelength channel
(band):  visible (0.65 um), water vapor (6.8 um), thermal IR (10.7 um),
etc.  All of this organization is done by choosing appropriate
'descriptor's and referencing the subsets by those descriptors.

I am guessing that you do, in fact, have data files from different
satellite channels.  I say this based on the following portion of your
listing:

goes8_1_1998_233_0015
goes8_2_1998_233_0015

It appears that the '1' and '2' refer to two different bands since the
date and time is the same.  So, based on this assumption, I will
discuss how one would create a dataset with two different descriptors.

As a review, we have:

o data files are in McIDAS AREA format

o data files are not named using McIDAS naming conventions

o the data files are all located in the directory:

  /usr/local/ldm/data/archive/chennon/233-243_1998

o there are two different channels represented by your files

The first thing we need to do is decide what to name this dataset.  The
name can be anything you like subject to a few restrictions:

o the group portion of the dataset name is limited to 8 characters

o the descriptor portion of the dataset name is limited to 12 characters

o you really should choose a name that is relevant to the data; this
  will help you remember what the dataset is supposed to contain

(The first two of these restrictions are discussed in the Remarks
section of the online and User Guide help for DSSERVE).

Let's suppose that we decide to call your dataset CHRIS.

Let's further suppose that the '1' in goes8_1_1998_233_0015 refers to
channel 1, or the 0.65 um Visible band, and that the '2' in
goes8_2_1998_233_0015 refers to channel 2, or the 3.9 um IR band.
Let's agree to name the descriptor for the band 1 GOES-8 imagery G8-VIS, and
the descriptor for the GEOS-8 band 2 imagery G8-39.

The DSSERVE commands that would define your datset would then be:

DSSERVE ADD CHRIS/G8-VIS AREA TYPE=IMAGE 
DIRFILE=/usr/local/ldm/data/archive/chennon/233-243_1998/goes8_1_* "Chris 
GOES-8 0.65 um imagery
DSSERVE ADD CHRIS/G8-39 AREA TYPE=IMAGE 
DIRFILE=/usr/local/ldm/data/archive/chennon/233-243_1998/goes8_2_* "Chris 
GOES-8 3.9 um imagery

(these are 2 lines, not 4)

NOTE: there are restrictions on the number and type characters that can be
used in the comment portion of DSSERVE commands:

o the comment must be less than or equal to 80 characters

o the comment contain an equal sign (=), apostrophe ('), or comman (,)

(This is described the online and User Guide help for DSSERVE).

If the CHRIS dataset was not previously defined, then the DATALOCation
for CHRIS will be set to be LOCAL-DATA by DSSERVE.  This is what you
want since you commented that you do not need to access the data from
other machines.

>When I run DATALOC currently, I get:
>
>Group Name             Server IP Address
>---------------         -----------------------
>BLIZZARD               ADDE.UCAR.EDU
>MYDATA                 <LOCAL-DATA>

OK.  After you run the DSSERVE commands above (or equivalents that match
how you want to setup things), your listing would look like:

Group Name              Server IP Address
---------------         -----------------------
BLIZZARD                ADDE.UCAR.EDU
CHRIS                   <LOCAL-DATA>
MYDATA                  <LOCAL-DATA>

>I assume <LOCAL-DATA> looks in $MCPATH for areas?

LOCAL-DATA means that McIDAS commands will not try to go out to a
remote machine to find that data.  Your question is a good one, but a
full explanation should wait until you become a little more experienced
with McIDAS (I don't want to scare you off, or is it too late already
:-).

>So does that mean I
>should move all of my area files to /home/mcidas/data?

No, absolutely not.  I strongly recommend that sites NOT put data files
in /home/mcidas/data.  The reason for this is that this directory gets
updated each time you do a new installation or update of McIDAS-X.

You use the DIRFILE keyword of DSSERVE to tell the McIDAS ADDE server
where to find data files that do not use McIDAS file naming
conventions.

>Or do I have to
>define new group names for each set of data I want to view?  

You can define new group names, or you can define new descriptors with
existing groups.  It really depends on how the data is logically organized.

>Basically, I'm just trying to figure out how to store and access my files.

I understand.  Did the above help or did it further muddy the picture?

>Can you clear up my confusion?  Thanks.

I hope so.  There are a number of additional things that I will probably
need to explain to you, but I want to wait until you can access the
data you already have (i.e., let's take this one step at a time).

Tom