19990820: decoding grib by XCD (cont.)

>From: "Jennie L. Moody" <address@hidden>
>Organization: UVa
>Keywords: 199907281535.JAA02200 McIDAS GRIB DMGRID

Jennie,

>so, we still don't know why the poke doesn't work, right?

Right.  This is a puzzler that deserves to be investigated.  It may
lead to some quick fixes that make life easier.

>I guess maybe you can login next Monday, I will try to find
>out from Tony if his cron jobs all worked....obviously, I
>just didn't want you on poking things as mcidas when he was
>running jobs...

OK.

>Sorry about this its our *#$%^#$ non-standard naming convention, aeolus
>still has the legacy of having been set up by Clay, he created a
>user called "ldmadmin, for LDM Administrator, 

This solves the mystery.

>so, all I meant was that I was (we continue to) intitate the
>decoding process by catting grib files into the spool with:
>
>cat somegribfile > xcd_run GRID (or HRS)
>this is done as _user_ ldmadmin, so that we have all the
>environment stuff etc. for running ingebin.k 

Actually, since the environmental stuff for running ingebin.k is contained
in xcd_run, you could run this as any user in the same group as ldmadmin
(assuming, of course, that you setup has the user ldmadmin, mcidas,
and mcadde (and gempak) in the same group).

>the grid monitor process dmgrid is started from within a mcidas
>session as user mcidas, before we were running startxcd, but recently
>we have just used the dmgrid command directly (after you told
>us about the pointer)....

OK, good.  If you are doing things by hand, there is no need to have
startxcd running.

>I still am pretty sure that it never
>read the pointer= line when I tried it, since the several times
>I had tried this commnad (this was a while ago) but seem to 
>be able to make it look at the top of a spool...

We need to get to the bottom of this as well.

re: writing output GRID files to an NFS mounted drive is slow
>Umm, its on a hard disk attached to aeolus, but its not on 
>an NFS mounted file system...so I don't think this is the problem.

My slant on this one was that one of the reasons that the decoding might
have been very slow could have been due to writing to an NFS mounted
file system.  If you are writing to a local disk, the problem falls
back to being that aeolus is just plain slow.

re: use ps or top to check process status
>I like top, but this doesn't seem much quicker or more obvious to
>me that just watching for the file to stop changing :-)

Well, you are correct.  One could, however, construct an automated checker
that would go out and do the cat of the grib data to ingebin.k based on
status output from ps.  I included the top output since it has a very
easy to read indication of the status of the process.  'ps' can give
you the same sort of thing, but not nearly so easily readable.

>Thanks for the pictures.
>
>Actually, I think this is what happens....and I don't think it matters
>if the spool wraps around as long as the decoder has stopped running,

You are correct UNLESS the wrap around goes past the existing start of
new data point.  The degenerative case would be where the grib product
was bigger than the queue.

>right, because it will look like this 
>
>                           +----------+
>                           +          +    
>                           +          + <- new data end here
>                           +          +
>                           +          +
>                           +          +
>                           +          + 
>                           +          + 
>                           +          + 
> dmgrid lass byte read ->  +          +           
>                           +          + <- new data starts writng here
>                           +          + 
>                           +----------+ 
>
>and it will just pick-up reading the new data, and will follow
>the spool around, right?

Right.

>The problem is, Tony was trying to tell me your pictures 
>couldn't describe the case when he had his problems, because 
>the spool hadn't even reached its maximum size yet....

If this is true, then the problem would seem to have to be related back
to the pointer in GRIBDEC.PRO.

>I haven't shown him/discussed your note, but I encouraged
>him to just make certain that he waited until the previous
>decoding had ended before he added to the spool....

OK, we need to cook up something better than having a person sit around
until a computer process finishes.  I will have to think about this
more later.

re: did pictures/description help
>yes, it was what I thought was happening, but I don't always trust
>my own thoughts on this stuff.

When it comes to McIDAS, it is easy to not trust one's own thoughts 
(some things in McIDAS are definitely counter intuitive).

re: multiple grids of the exact same field being decoded into the
same GRID file being normal
>Okay...good.

re: go to the head of the class
>hmmm....can't tell if you are making fun of me
>for stating the obvious....

I was not making fun of you at all.  Your comments indicate that you
do know a lot about what is going on.  Sometimes you just don't quite
know enough.  This is not your failing, but, rather, an indication of
how hard McIDAS is to really master.  This is for those trying to use
it.

re: understanding what is going on
>well, thats a suprise.

There are unfortunately very few in the Unidata McIDAS community that have
a good grasp of the power of McIDAS.  The real problems are ones of
a lot of historical baggage and semi-infinite flexibility.  The flexibility
becomes more and more useful/desirable as one gets to be more and more
of an expert in McIDAS.  For novice or casual users, it is a killer.

>I don't feel like I understand anythings-Unidata(mcidas,ldm, etc) well...
>For instance, I just pulled down the new mcidas7.6,
>it needs netCDF (which I thought we might already have, but I guess
>we don't, so I need to get it, and figure out where to put it), 

No!  McIDAS-X 7.6 comes bundled with netCDF.  You do NOT have to go out
and get it separately.  It will be built for you during the McIDAS
make phase.

>I need to get the new ldm-mcidas to go with 7.6,

Yes, but I recommend that you grab a binary distribution.  This way all
you have to do is unpack the distribution and copy the new executables
(the ones you use in any case) into the directory where you have
your current ldm-mcidas decoders.

>this got me
>looking at the ldm, and I don't have the latest version of this, so
>I should really get the ldm upgrade...

What version do you have?  If you have 5.0.6, then holding off on an
upgrade is no big thing.  If you are back at something like 5.0.1, then
you should upgrade.  Versions inbetween the two above are a toss up.

>and as long as I'm going to
>upgrade it, I should put it in the standard location of /home/ldm

The standard installation for the LDM is in /usr/local/ldm.  The
standard name for the ldm user is 'ldm'.  There is nothing wrong with
/home/ldm, however.

>(currently on windfall its in /home/ldma, cause Kevin used and adduser
>tool developed here at UVA, but it won't allow 3-letter userids,

You could use the Solaris 'admintool' to do your user administration.
This GUI is nice and easy to run, so it is not a big thing to pick up.

>so I need to figure out how to add a user without the tool)....

Check out admintool, or simply edit the /etc/passwd file directly.
I suggest using admintool.

>more
>than I want to deal with (since I should be analyzing data, writing
>up results, etc)...

I don't blame you.

>but, enough whining:  
>I have come up with the following order, 
>does this sound right?
>
>       o get the netcdf binaries (and figure out where they go)

No, you don't need them since the netCDF comes bundled with McIDAS-X.

>       o build the new mcidas7.6
>       o test the new mcidas7.6
>       o find all the unique pieces of 7.4
>          (which we did not conveniently put in a
>           subdirectory, or rename uva_)
>          and add them (modify the makefile)
>          to 7.6

Right.  While you are doing this, I recommend moving all of those files
that you modify from the ~mcidas/data directory to the ~mcidas/workdata
directory.  BATCH and McBASI scripts in ~mcidas/workdata do not get
overwritten by new installations.  As far as your own programs go, then
you would have to copy them over to the mcidas7.6/src directory and
modify the makefile, or you could create a ~mcidas/localsrc (name it
whatever you wish) and put them in there.  You would then need to
generate a makefile so you could build/rebuild them when you install a
new distribution.  I recall recommending you NOT do this some time ago,
but I now realize that you would be better off in the long run by doing
this.  With your own source in a separate directory, you would not have
to worry about it being overwritten by a new distribution.  With your
own makefile, you would not have to muck with the makefile in the new
McIDAS distribution.

>       o re-make these parts in 7.6 
>       o backup everything in the old mcidas/data
>          (and workdata?) direcectories (batches and 
>          scripts)

BATCH files and McIBASI scripts in workdata will not be overwritten
(except mcscour.sh).  BATCH files and McBASI scripts in data will
be overwritten if they have the same name as BATCH files/McBASI scripts
in the new distribution.

>at this point, I guess I could stop the ldm, and un-install 
>the old mcidas version and install the new one, *but* I think
>I have to install the new ldm-mcidas first...right?

Since there have been schema changes along the way (to make the dates
in MD files Y2K compatible), it is best to upgrade ldm-mcidas decoders
and McIDAS-X,-XCD in lock step.  Since some of the schema changes will
change the "standard" MD files (e.g. surface, upper air, synoptic, etc.),
it is best to do the changeover right around (just after) 0Z.  The idea
here is that you will need fresh MD files to decode into after installing
7.6.  If you do the switchover after one of the "standard" MD files has
already been created by the 7.4 version of the XCD decoder, then the
procedure is:

o stop the ldm
o delete the MD file(s) for the current day
o uninstall the previous McIDAS version
o install the current McIDAS version
o make sure that you copy SCHEMA from the 7.6 distribution to the directory
  where XCD decoders create their output MD files; note that if you keep
  XCD produced MD files in a directory that is separate from the MD files
  created by ldm-mcidas decoders, then you will need  to either copy or
  link SCHEMA to the ldm-mcidas output directory as well (I don't think
  you are in this situation, but others reading this message will need
  to understand that this is the case)

>       o stop the ldm
>        o as long as I am getting a new ldm-mcidas, might
>          as well, make a new user (really make it ldm)
>          and get the new binaries for ldm and ldm-mcidas
>          both, and put both of these into the /home/ldm
>          account, then go through configuring...this will
>          be as if for a first time...

If you are going to standardize your system, then I recommend installing
the LDM in /usr/local as per the online LDM instructions.

>       o finally, go back and install the new mcidas
>          (since the intervening steps may take me a little
>          while, I don't want to change the mcidas version
>          to soon....also, if we are not concerned about
>          accessing real-time data, can we keep the old version
>          of mcidas around for running (maybe keep the old
>          .variables.ksh file which would make all the 
>          environment variables point to old directories..
>          or is this really dangerous....I just don't want 
>          to discover we have problems with using our 
>          scripts to process water vapor imagery under
>          7.6 since I have to prepare a presentation
>          for mid-September, and I don't want any nasty
>          surprises of things we hadn't anticipated not working...)

OK, if you have the disk space, then you may want to alter the McIDAS
installation to be even more flexible.  Since I have to build McIDAS
on 7 different operating systems AND keep around the last two distributions
so I can investigate user problems, I adopted the LDM method of supporting
diffent versions.  What I do is:

o instead of setting my McIDAS installation directory (defined in
  the McINST_ROOT Unix environment variable) to be /home/mcidas
  (the standard, recommended way), I set it to be a directory whose
  name reflects the distribution revision level.  For McIDAS-X 7.50
  this was mcidas75; for McIDAS-X 7.60 this is mcidas76.

o with the above definition of McINST_ROOT active, I make the new
  McIDAS distribition:

  <login as 'mcidas'>
  export McINST_ROOT=/home/mcidas/mcidas76
  cd mcidas7.6/src
  make all             <-- build -X and -XCD in one step
  <test the distribution after the build finishes>
  make install.all     <-- this installs under /home/mcidas/mcidas76

o in the home directory for the user 'mcidas', create a series of
  links:

  cd /home/mcidas
  ln -s mcidas76         runtime
  ln -s runtime/admin    admin
  ln -s runtime/bin      bin
  ln -s runtime/data     data
  ln -s runtime/help     help
  ln -s runtime/inc      inc
  ln -s runtime/lib      lib
  ln -s runtime/man      man
  ln -s runtime/savedata savedata
  ln -s runtime/tcl      tcl
  ln -s runtime/workdata workdata

o the environment under which you operate would have McIDAS environment
  definitions in the standard way:

  MCDATA=/home/mcidas/workdata
  MCPATH=${MCDATA}:/home/mcidas/data:/home/mcidas/help
  MCGUI=/home/mcidas/bin
  etc.

o when you build a new McIDAS distribution, it will be put in a separate
  directory structure as you see above.  In order to use the new distribution
  for all things (McIDAS-X session, ROUTE PP BATCH files from the LDM, etc.)
  all you have to do is change the runtime link:

  cd /home/mcidas
  rm runtime
  ln -s mcidas74 runtime

  This makes the 7.4 version "active"

  cd /home/mcidas
  rm runtime
  ln -s mcidas76 runtime

  This makes the 7.6 version active

With this kind of structure, you can very quickly change between versions
by changing one soft link.

Now, in order to keep the XCD decoding stuff working, you might want
to make a different "workdata" directory that will be used for XCD
decoding (through xcd_run), ROUTE PostProcesses (through the Bourne shell
script batch.k that you have copied to somewhere in the 'ldm' directory
tree.  What I do is:

o cd /home/mcidas
  mkdir upcworkdata
  cp workdata/* upcworkdata

  This replicates all of the files in workdata in upcworkdata.  The
  /home/mcidas/upcworkdata directory is then my MCDATA directory in
  xcd_run, batch.k, and mcscour.sh.  The MCDATA directory defined
  in 'mcidas's .profile (or .kshrc, or .bashrc, or .cshrc, etc.)
  and .mcenv (used by the ADDE remote server) would point to
  /home/mcidas/workdata.

The drawback with leaving the pointing of MCDATA to
/home/mcidas/workdata is the need to make sure that the datasets get
defined when you do the new installation.  You could have MCDATA
defined in .mcenv to be /home/mcidas/upcworkdata (whatever you want to
call this separate directory) so that the ADDE remote server will not
have to be reconfigured upon each new installation.

The beauty of this setup is that you can install new versions of
McIDAS without worrying about the new distribution overwriting the
setup for the previous.  The bad thing about this is that you have
to remember to update files in the "upcworkdata" directory "by hand"
unon the new install.

Like I said in the previous email, there are typically multiple ways
to do the same thing in McIDAS.

>Sorry for the long list....And I have been reading the documentation,
>but since I have to do several upgrades, its tricky figuring out
>the order.

No problem.

>Thanks for your help!

You are welcome.

re: multiple spool files
>I like this idea a lot, this stuff would be a lot faster on windfall,
>and we have a lot more disk space. But it has to come after the
>upgrades.

Waiting until after the upgrade AND when you have some time is a good
idea.

Later...

Tom