20001007: confusion about setting up ADDE datasets

>From:  "Jennie L. Moody" <address@hidden>
>Organization:  UVa
>Keywords:  200010080108.e9818i415729 McIDAS-X ADDE

Jennie,

>I am trying to access data that was archived (on windfall) using the
>dell from home.

OK.  This will be a good interchange for the archives since it is the
sort of thing that a number of users will want to do.  Please forgive
additional comments in my reply.  I am adding them so that others reading
along will have a broader view of what is needed.

>So, we have
>defined a bunch of group/descriptor dataset, TOPSEA, TOPSEB and TOPSEC
>for example.  So, I took the following steps, and now I am confused:

Just to be precise for the archives, TOPSEA, TOPSEB and TOPSEC are
group names of datasets.  Under these will be defined types of data.

>1)I edited the LOCDATA.BAT file and added a line to access the TOPSEC
>  archived data:
>
>DATALOC ADD TOPSEC windfall.evsc.virginia.edu

Excellent.  Just for archive reference, the machine may be specified
either by name (as you have done) or by IP address.  Also, capitalization
is not important in the machine name.

>2)I ran a batch file that I copied over from windfall, it sets up the
>  dsserve info telling where the named dataset/descriptors are defined
>  (what AREA and GRID number ranges), eg:
>
>DSSERVE ADD TOPSEC/GE-IR AREA 5001 5999
>DSSERVE ADD TOPSEC/GE-WV AREA 6001 6999
>DSSERVE ADD TOPSEC/GE-AW AREA 7001 7999
>DSSERVE ADD TOPSEC/GE-R2 AREA 8001 8999
>
>DSSERVE ADD TOPSEC/ETA   GRID 1001 1999
>DSSERVE ADD TOPSEC/MRF   GRID 2001 2999
>DSSERVE ADD TOPSEC/RUC2  GRID 3001 3999

You are doing this on your client machine, not windfall?  You need to
define the dataset composition on the machine that will be serving the
data, not the client.

Since you are setting up the DATALOC to "point" at windfall when you
ask for TOPSEC data, you do not need to define the datasets on your
client.  You do need to define the datasets on the server.

>3)This is where I get confused, when I do this on windfall, the last
>  step in defining where the data are located is with a set of file
>  redirections in a .NAM file.  I copied over this file, with entries
>  like:
>
>AREA1* /q4/links/topseC/images/goes10ir
>AREA2* /q4/links/topseC/images/goes10wv
>AREA3* /q4/links/topseC/images/altw.mrf
>AREA4* /q4/links/topseC/images/altw.ruc2
>AREA5* /q4/links/topseC/images/goes8ir
>AREA6* /q4/links/topseC/images/goes8wv

Again, unless the images are going to be on your client, this is not
necessary.

By the way, it doesn't look like you did the DSSERVE steps on windfall.
If you had, there would be entries in the copy of RESOLV.SRV that is
located in the working directory for your 'mcidas' user (this is
uvaworkdata on your machine and workdata on other users' machines).

Also, I want to warn you about the names of AREA files you have
chosen.  I send out example REDIRECTions using AREA10*.  This range of
images is used by sites that are running the ldm-mcidas decoder,
nids2area (AREA10*) AND by sites that are saving the CIMSS imagery
(AREA11*) that was added in June, so there is a potential name clash.

The REDIRECTions that are already in place in your 'mcidas' account
on windfall are more specific than AREA1*, so they will be used
in preference to your REDIRECTion entry above.

I.e.:

AREA10* /p4/data/mcidasd
AREA11* /p4/data/mcidasd

are more specific than:

AREA1* /q4/links/topseC/images/goes10ir

The more specific a REDIRECTion is, the higher in the search hierarchy
it will be.

Your options are to change the range of AREA file numbers you want to
use for your TOPS* datasets

AREA12* /q4/links/topseC/images/goes10ir

or change where the CIMSS images are being written to on windfall (you
don't have to worry about the AREA10* file names since you are not
running nids2area).

The latter option would require that you change the routing table entries
for the CIMSS products, so it would be a bit messier.

I hate to say this, but all of this will get a lot easier with the
7.7 version of McIDAS, and I am simply not ready to release it yet
(I feel _really_ bad about this, but ...).

>But, (and I suppose its obvious) this doesn't work when I set the
>redirections this way on my home machine.  At first, I thought it
>might, thinking it uses the redirection information to get the right
>directory, etc, for the files in the ranges defined by dsserve and
>located (on the server windfall) with the LOCDATA.  However, now I find
>myself wondering if it even uses the file redirections on this (the
>client) end.

You came up with the correct conclusion.  The information on your client
machine (your home machine) is not used at all for a dataset on a remote
machine.  The best way to remember this is the following: as a user,
you should have no idea of where the data is located on the server, or
what files the data is in (i.e., their names).  All you care about
as an ADDE users is:

o the GROUP name of a dataset
o the machine the dataset is being served by
o whether or not you are allowed by the server to access the data

>It doesn't seem to make sense that this is how the files
>get found.  I know that you explained to me how the data on the remote
>server gets accessed (what steps occur) once before, but I dug around
>and couldn't find it.  Since this is sort of a different instance I'm
>not sure that answer would explain this problem anyway (you had
>explained to me how it found real-time data on a remote server), so
>maybe this is different?

What you need to do is logon to windfall as 'mcidas' and setup the
datasets you want. This involves:

o identifying the data to be organized by dataset GROUP and TYPE name
  (i.e., figure out what you want to serve and what you want to call it;
  you have pretty much done this except for the AREA file name clash
  I discussed above)

o setting up the dataset with DSSERVE commands (like you show above)

o making sure that the ADDE remote server user, 'mcadde', can find the
  data files in the datasets (this is the adding of REDIRECTions
  discussed above)

Notice that all of a sudden I mention the user 'mcadde'.  Remember that
'mcadde' is in essence nothing more than an alias for the user 'mcidas'.
'mcadde' shares the same HOME directory, the same configuration files;
etc.; as 'mcidas'.  The big difference is that you can not login as
'mcadde' since its default shell should have been set to /bin/false.
  
>I hate asking open-ended questions, and this may sound really fuzzy to
>you.

It doesn't sound fuzzy at all.  What is missing is your understanding
of the dual roles involved:

o the first is as a data "publisher"

o the second is as a data "user"

The publisher activities are done on the server; the user activities
are done on the client.  (The fact that the client machine could be
the same as the server machine is what might be confusing things).

>If this is going to require you to type a long reply that makes you
>feel like you are repeating yourself, I apologize.  Maybe I am just too
>tired to figure this out right now.  But on the chance that its an easy
>answer, I'll just ask anyway.

The answer is easy.  You need to define the datasets (do the DSSERVE
invocations) as the user 'mcidas' on the server machine,
windfall.evsc.virginia.edu.   You must also define the locations of
the data files that comprise the datasets on the server.  This would
be through REDIRECTins or through MCPATH.

All you have to do on the client is define where to look when asking
for the dataset:

DATALOC ADD TOPSEC windfall.evsc.virginia.edu

I think that it will be best for you to work through setting up these
datasets with help if/when necessary.  Once you have done one, the
process will become easier and easier.

Tom


>From: "Jennie L. Moody" <address@hidden>
>Date: Sun, 8 Oct 2000 15:41:14 -0400 (EDT)

re: You are doing this on your client machine, not windfall?  You need to
define the dataset composition on the machine that will be serving the
data, not the client.

>Okay, I see the difference I guess.  These datasets are already
>defined on windfall, and they have been saved in these GRID
>and AREA ranges.  I guess when I invoke them on windfall (which
>in that instance is both the server and client) I have a DATALOC
>command that adds them as LOCAL DATA.

re: Since you are setting up the DATALOC to "point" at windfall when you
ask for TOPSEC data, you do not need to define the datasets on your
client.  You do need to define the datasets on the server.

Right.  This is the conclusion I was coming to.

re: By the way, it doesn't look like you did the DSSERVE steps on windfall.
If you had, there would be entries in the copy of RESOLV.SRV that is
located in the working directory for your 'mcidas' user (this is
uvaworkdata on your machine and workdata on other users' machines).

>Okay, right, I had done them on windfall, but as me (a user, jlm8h)
>not as mcidas.  And, now I can see the looming conflict with the
>way we have defined files (AREAS, GRIDS, etc).  See, this isn't
>something I am in the act of defining now, these are structures we
>put in place when we built the archive (ie., during the experiment
>last spring).  To compound matters, we then ran another experiment
>over the summer, and continued to use the same structure, just
>writing to different physical disks. This works fine, ie., without
>conflict, as long as we access things from user accounts on windfall,
>because we never have to ask the user McIDAS to do anything (and I
>don't want to conflict with the realtime actions of McIDAS).  

>I guess we don't have any nids data anyway right now, and I am not
>sure if we are using any of the CIMSS derived image products anyway
>right now (we would if I was teaching but I'm not).

re: AREA file names in your archive

>So, one solution is to make a number of more specific AREA and
>GRID redirections that will allow for the access to archived data.
>But, since at different times, we want to access different
>data-archives, we will have to dynamically change the user mcidas
>information regarding what datasets we want to serve.
>This seems pretty non-ideal, doesn't it?  My view at the moment would be
>that this kind of a client/server model limits the ability of
>a server to include a lot of archived datasets.  

re: or change where the CIMSS images are being written to on windfall (you
don't have to worry about the AREA10* file names since you are not
running nids2area).

>Even if I did this, if you understand my dilemma, I have another
>project (PROPHET), which also generated an archive of GRIDS and
>AREAS, using the same naming structures, etc. which we used
>for the TOPSE experiment.  We just wrote to different real-locations.
>As long as we control access to these locations (as we do with
>our own set of USER redirections on windfall) this works fine,
>but I guess its going to screw up remote access from home.

>After writing this message, I will see if I can make (carefully) a
>couple of changes to the user mcidas information to allow me to
>see at least one part of the archives (just to test my understanding
>of what you have said).  Then, I will try to see how much conflict
>exists, or if its just potential conflict (we don't get nids right
>now, and I cannot tell you off the top of my head if we are archiving
>the CIMSS products or not, I think we may not be).

re: o making sure that the ADDE remote server user, 'mcadde', can find the
data files in the datasets (this is the adding of REDIRECTions
discussed above)

>Right, I will check this out, as noted, these archives already exist, 
>and I have been accessing them from windfall.

>Well, from the way it looks, I have these nice big
>archives set up (raw and derived images, grids, etc.)
>that now I cannot access without messing up the realtime
>data access, since it will affect the set of file
>redirections used by user-mcidas.  

>This is a painful realization, since the whole point
>of having mcidas running from home was NOT to look at
>real-time weather, but to begin to analyze (in earnest)
>data collected over several months of the past spring.

>This is a *lot* of data to rename, and or, a complicated
>job to figure out how to avoid any conflicts between
>the real-time and archived data.

>Thanks, I am now less confused (but much more discouraged).

>Jennie

>From: Tom Yoksas <address@hidden>
>Date: Sun, 08 Oct 2000 20:29:32 -0600

Please read through the entire message 'cause I have a recommendation
near the end.

>Okay, I see the difference I guess.  These datasets are already
>defined on windfall,

But, not through the 'mcidas' account.

>and they have been saved in these GRID
>and AREA ranges.  I guess when I invoke them on windfall (which
>in that instance is both the server and client) I have a DATALOC
>command that adds them as LOCAL DATA.

I see from below that this is through your account, not the 'mcidas'
account.

re: You do need to define the datasets on the server.

>Right.  This is the conclusion I was coming to.

OK, so we are on the same page.

re: DSSERVEs have not been run in the 'mcidas' account

>Okay, right, I had done them on windfall, but as me (a user, jlm8h)
>not as mcidas.  And, now I can see the looming conflict with the
>way we have defined files (AREAS, GRIDS, etc).  See, this isn't
>something I am in the act of defining now, these are structures we
>put in place when we built the archive (ie., during the experiment
>last spring).  To compound matters, we then ran another experiment
>over the summer, and continued to use the same structure, just
>writing to different physical disks. This works fine, ie., without
>conflict, as long as we access things from user accounts on windfall,
>because we never have to ask the user McIDAS to do anything (and I
>don't want to conflict with the realtime actions of McIDAS).  

Right.  From the 'mcidas' account there is now a naming conflict.

>I guess we don't have any nids data anyway right now, and I am not
>sure if we are using any of the CIMSS derived image products anyway
>right now (we would if I was teaching but I'm not).

Right, you aren't running the ldm-mcidas nids2area decoder to
create AREA files from the NIDS products.  As far as the CIMSS products
go, you are not using these either yet, but you may want to at
some point.  Especially the Ozone product.

>So, one solution is to make a number of more specific AREA and
>GRID redirections that will allow for the access to archived data.
>But, since at different times, we want to access different
>data-archives, we will have to dynamically change the user mcidas
>information regarding what datasets we want to serve.
>This seems pretty non-ideal, doesn't it?

Exactly.  This was the HUGE drawback in the McIDAS AREA file naming
convention.  One had to plan to not cause name conflicts.  This is
one of the things we fought SSEC about for years.  McIDAS-X 7.7
_finally_ gives us all a way to get around this nonesense.

>My view at the moment would be
>that this kind of a client/server model limits the ability of
>a server to include a lot of archived datasets.  

My feelings exactly!  It was a long, hard fight to get them to realized
(or act on) this insanity.

re: Your options are to change the range of AREA file numbers you want to
use for your TOPS* datasets
AREA12* /q4/links/topseC/images/goes10ir

>Note, these are already defined, sitting out on a mass-store
>(which is what the /q4/links....accesses).

OK, so renaming files is really not an option.

re: renaming

>Even if I did this, if you understand my dilemma, I have another
>project (PROPHET), which also generated an archive of GRIDS and
>AREAS, using the same naming structures, etc. which we used
>for the TOPSE experiment.  We just wrote to different real-locations.
>As long as we control access to these locations (as we do with
>our own set of USER redirections on windfall) this works fine,
>but I guess its going to screw up remote access from home.

With Version 7.6x yes; with 7.7x no.

Installing a new McIDAS distribution is pretty easy. The hard part would
be integrating your site code into the source tree and makefile.  We
need to redo this so life will be easier in the future.

You are receiving the CIMSS products.  I know this because I set it
up for you.  These can be easily turned off, however.  The change
is a simple one in the LDM pqact.conf file and a HUP sent to pqact.
This doesn't get around the multiple datasets with the same set
of file names problem, however.  The changes in how one can setup
ADDE datasets in 7.7, however, does address this.

Here goes with the recommendation;

If you have enough disk space, we could do the following:

o change the HOME directory for 'mcadde' to be /home/mcadde
o copy ~mcidas/.mcenv to ~mcadde
o build McIDAS-X 7.7 in ~mcadde
o configure the 7.7 release to be able to access your datasets and
  the realtime data
o eventually, the UVa additions to McIDAS would need to be added
  to 7.7.  I recommend, however, that this be done in a directory
  other than the mcidas7.7/src directory.  I would recommend creating
  a ~mcadde/uvasrc directory and copying all of the UVa code to it.
  A new Makefile would have to be created so that your site-specific
  stuff would be easily built/modified.  The executables from these
  source files could be installed either in a separate 'bin'
  directory (e.g., ~mcadde/uvabin) or in the ~mcadde/bin directory.
o eventually, after things have settled down, /home/mcidas would
  be replaced by /home/mcadde

This setup (not entirely defined in the above list) would allow the
realtime ingestion and archiving activities to proceed AND for you
to access the other datasets.  It does put off the inevitable remerging
of processes, but it will give you breathing room.

Please let me know if this is acceptable to you.

>From: "Jennie L. Moody" <address@hidden>
>Date: Sun, 8 Oct 2000 17:52:50 -0400 (EDT)

re: access to the nice big archives

Installing 7.7 would allow this.

>This is a painful realization, since the whole point
>of having mcidas running from home was NOT to look at
>real-time weather, but to begin to analyze (in earnest)
>data collected over several months of the past spring.

This would be possible.

>This is a *lot* of data to rename, and or, a complicated
>job to figure out how to avoid any conflicts between
>the real-time and archived data.

Forget the renaming.  This is made unnecessary by installation of
7.7.

>How could I have failed so miserably to foresee this
>difficulty?

This is simply a failing of pre-7.7 versions of McIDAS.  Nobody
should have to worry about this kind of crap.

>Thanks, I am now less confused (but much more discouraged).

Again, let me know.

Tom

>From: "Jennie L. Moody" <address@hidden>
>Date: Mon, 9 Oct 2000 15:17:10 -0400 (EDT)

re: CIMSS image products in Unidata-Wisconsin

>Actually, we have been getting these data as MD files (not
>AREA's) from Chris Schmidt at Wisconsin for some of our work
>for the TOPSE campaign.  But, yes, we would like to use the Ozone
>product.

re: OK, so renaming files is really not an option.

>Good, I didn't want to mess around with this, and in fact, if
>I had to do it this way, what had occurred to me was to
>pull over chunks of the data out of the archive and just keep
>them as local datasets on the Dell.  Obviously, I cannot move
>the whole archive over here, but I guess I could think hard about
>what I want to do most asap and grab these data, loading them into
>parallel named directories on the dell (barb).

re: Installing a new McIDAS distribution is pretty easy. The hard part would
be integrating your site code into the source tree and makefile.  We
need to redo this so life will be easier in the future.

>Right.  Well, is 7.7 really ready to go, even in a beta-test version?

re: setup 7.7 under 'mcadde' account

>Okay, I think I get this.  Is  there any incompatibility if the
>version of McIDAS that is making real-time imagery (7.6) is not
>the same version of McIDAS that is serving imagery (7.7)? 

>The reason for doing this is that then McIDAS7.7 would be controlling
>all the server commands?  

>Are you pretty sure we won't create any conflicts by having two
>versions of McIDAS installed?  

>I think we have plenty of disk space.  

>So how does the new version 7.7 work differently in allowing one
>to specify the location (and naming) of data available on the 
>server?

>To: "Jennie L. Moody" <address@hidden>
>Date: Mon, 09 Oct 2000 17:21:15 -0600

>Right.  Well, is 7.7 really ready to go, even in a beta-test version?

Yes.  I have been running it here since June, and my distribution is
working well on all of my supported platforms.  What I do not have
done is the GUI, and it looks like this will not be done before
the workshop.

re: CIMSS setup

re: loading in McIDAS-X 7.7 to give you ADDE access at home

>Okay, I think I get this.  Is  there any incompatibility if the
>version of McIDAS that is making real-time imagery (7.6) is not
>the same version of McIDAS that is serving imagery (7.7)? 

There shouldn't be.  I did find a modification in 7.7 that had problems
getting data via ADDE for getting datasets from 7.6 servers, but I
put the code fix into a 7.6 addendum.

>The reason for doing this is that then McIDAS7.7 would be controlling
>all the server commands?  

7.7 would be the remote server, yes.  Its code would be used only for
serving data until you have a chance of merging in your local code, etc.
I don't anticipate any big problems.

>Are you pretty sure we won't create any conflicts by having two
>versions of McIDAS installed?  

You won't as long as they are in different accounts.  This is why
I recommend installing 7.7 under the newly created home directory
of the user 'mcadde'.  This would break the paradigm that 'mcadde'
is only an alias for 'mcidas', but this is not a problem.  Again,
things should be fused back together as things settle down.

>I think we have plenty of disk space.  

Not in /home.  I took a quick look on windfall, and I did not see
enough disk space.  I note, however, that there is both the 7.6 and
7.4 versions of McIDAS-X under /home/mcidas.  I think that there
will be enough room if the 7.4 stuff is removed.

>So how does the new version 7.7 work differently in allowing one
>to specify the location (and naming) of data available on the 
>server?

7.7 allows one to define datasets for files of McIDAS form but not
named using the old McIDAS conventions (e.g., AREAnnnn, MDXXnnnn, 
GRIDnnnn).  An example DSSERVE command that sets up a dataset
will probably explain things.

Let's use the example that you already sent in:

2)I ran a batch file that I copied over from windfall, it sets up the
  dsserve info telling where the named dataset/descriptors are defined
  (what AREA and GRID number ranges), eg:

DSSERVE ADD TOPSEC/GE-IR AREA 5001 5999
DSSERVE ADD TOPSEC/GE-WV AREA 6001 6999
DSSERVE ADD TOPSEC/GE-AW AREA 7001 7999
DSSERVE ADD TOPSEC/GE-R2 AREA 8001 8999

DSSERVE ADD TOPSEC/ETA   GRID 1001 1999
DSSERVE ADD TOPSEC/MRF   GRID 2001 2999
DSSERVE ADD TOPSEC/RUC2  GRID 3001 3999

AREA1* /q4/links/topseC/images/goes10ir
AREA2* /q4/links/topseC/images/goes10wv
AREA3* /q4/links/topseC/images/altw.mrf
AREA4* /q4/links/topseC/images/altw.ruc2
AREA5* /q4/links/topseC/images/goes8ir
AREA6* /q4/links/topseC/images/goes8wv

The location of the AREA1* files is /q4/links/topseC/images/goes10ir.
The names of the files are AREA1*.  The DSSERVE command that can be
used to setup this dataset GROUP/TYPE is:

DSSERVE ADD TOPSEC/GE-IR AREA DIRFILE=/q4/links/topseC/images/goes10ir/AREA1* 
"TOPSEC 10.7 um images

You can see that you can now specify a regular expression for the location
and name of the files that comprise a dataset GROUP/TYPE.  The fact that
you do not use REDIRECT or MCPATH to locate the files makes it possible
have multiple datasets defined using the same file names, but located
in different directories.  This will make your life easier!

>I guess I will work on some other things (there is no shortage
>of work here) until you let me know when to proceed (I presume
>7.7 isn't sitting out there publicly available for download).

Right, it is not sitting out there ready for public download.  I can
cut a distribution tomorrow and get things moving IF you are ready
(meaning that the 7.4 McIDAS-X installation in the 'mcidas' account can
be wiped out with fear of losing something important.  Since I don't
know what you really have going on, I can't start diving off making
changes.

>Thanks Tom, for all your help.

No problem.  What else do I have to do :-)

Tom