20021014: Setting up LDM for McIDAS-XCD

>From: Richard Massa <address@hidden>
>Organization: UC Davis
>Keywords: 200210150109.g9F19F112387 McIDAS-XCD LDM

Hi Richard,

>Hello,  I'm the new sysadmin for the UC Davis Atmospheric Sciences group,
>replacing Erick Lorenz,

Did Erick leave UC Davis (just curious)?

>and I've taken over the adminstration of the LDM
>system, which wasn't (according to my professors) in a working state.  So I've
>set up a new system running the latest version of the LDM software and I've
>been having some problems...

Sounds good.

>I guess the main thing that I need to ask for is help with pqact (or maybe a 
>sample configuration file) that places the products into the correct locations
>for an ADDE server.

OK.  There are three packages that come into play here:  LDM,
ldm-mcidas, and McIDAS-XCD.  Both ldm-mcidas and McIDAS-XCD are
decoders for products that are ingested by the LDM.

>My initial goal to set up the LDM program so the computer
>that runs it can function as a McIDAS ADDE server, that way the machines that
>run McIDAS don't have to connect to servers outside our network.

Sounds good.  What kind of a machine is this (operating system, etc.)?

>I'm currently feeding IDS, DDPLUS, UNIWISC, and HDS products off of
>peridot.atmos.ucla.edu, but I'm also guessing that I need at least the nexrad 
>and NLDN products in addition to what I have now.

The NEXRAD products (Level III products for individual NEXRADs) and
FNEXRAD products (national composites for Level III base reflectivities
and 1-hour and storm total precipitation) are very useful.  They can be
gotten from most upstream LDMs.  The NLDN data can only be gotten by a
point-to-point feed from SUNY Albany.  If UC Davis never got the data,
you will have to send a note to Dr. David Knight requesting a feed.  If
you previously got a feed but your machine changed, you will also have
to send David an email.  If you previously got a feed and you are using
the same machine in this incarnation of the LDM (same machine name and
IP address), you should be set to go.

>I've currently installed the LDM (5.1.2),

The current version of the LDM is 5.2.1, but it is not available in
binary format.

>McIDAS 2002a, and the ldm-mcidas and unidata decoders on the machine.

Sounds good.

>Is Gempak necessary to have in order for the
>ADDE server to function correctly?

No.  GEMPAK is not needed at all for McIDAS-related things.

>I see mention of in different places in the DSSERVE.BAT file.

I guess I was not explict enough with my comments in DSSERVE.BAT. The
idea is that GEMPAK requires the user to store data in a very specific
directory hierarchy.  McIDAS is flexible (to the extreme), so it can
use that hierarchy with ease.  This allows sites running both packages
to setup data filing for use with GEMPAK and McIDAS.

>Lastly, does the problem with newer kernels and the LDM not working as an ADDE
>server still exist or has it been fixed?

The problem that you are referring to is serving sounding data through
the remote ADDE server on Linux machines.  The problem has not been
seen on any other operating system that I support McIDAS on (and that
includes a variety of PC-UNIX variants).  Also, the problem has nothing
to do with the LDM itself.

Here goes with an overview of what you need to do to get the LDM setup
so that ldm-mcidas and McIDAS-XCD can produce data files that you then
can use for an ADDE server at your site.

For reference, the setup of the ldm-mcidas decoders -- including needed
LDM pqact.conf entries -- is discussed in the ldm-mcidas web pages:

http://my.unidata.ucar.edu/content/software/mcidas/mcidd/index.html

Example pqact.conf entries for ldm-mcidas decoders are found in:

http://my.unidata.ucar.edu/content/software/mcidas/mcidd/ldm-mcidas-use.html

Discussion of setting up LDM ldmd.conf and pqact.conf entries for McIDAS-XCD
use can be found in:

McIDAS-X HomePage
http://my.unidata.ucar.edu/content/software/mcidas/index.html

Installation and Configuration
http://my.unidata.ucar.edu/content/software/mcidas/2002/mcidas-x.html

Build, Install, and Configure XCD
http://www.unidata.ucar.edu/packages/mcidas/2002/mcx/mcidas-xcd.html

Unidata McIDAS-XCD Running with the LDM
http://www.unidata.ucar.edu/packages/mcidas/2002/mcx/xcd_start.html


The "quick and dirty" overview of how things work is:

1) the LDM on the local machine is used to get data from upstream LDMs

2) the LDM startup file, ~ldm/etc/ldmd.conf, must be modified to start
   a McIDAS-XCD startup shell script whose name is xcd_run.

   xcd_run is distributed with Unidata McIDAS.  It can be found in the
   ~mcidas/bin directory _after_ McIDAS has been installed.

   xcd_run MUST be copied to a directory in the PATH of the user running
   the LDM; typically this is a directory named ~ldm/decoders.

   the copy of xcd_run copied to the ~ldm/decoders directory MUST be
   edited and modified to set various enviornment variables to match
   how McIDAS was installed (mainly, the variable MCHOME).

   permissions for xcd_run MUST be set so that it is executable by
   the user running the LDM

   ~ldm/etc/ldmd.conf MUST be edited and changed to include the XCD
   startup sequence:

   exec 'xcd_run MONITOR'

3) the LDM pattern-action file, ~ldm/etc/pqact.conf, MUST be modified
   to include patterns for McIDAS-XCD execution.  These are _very_
   simple.  They look like:

   # Entries for XCD decoders
   DDPLUS|IDS   ^.*     PIPE
        xcd_run DDS
   HRS  ^.*     PIPE
        xcd_run HRS

   WARNING: entries in ~ldm/etc/pqact.conf MUST NEVER have leading spaces.
   The must also contain tabs as white space in various locations.  In
   the listing above, tabs and no spaces are found inbetween the IDS and
   ^.*; between the ^.* and PIPE; before 'xcd_run DDS'; between HRS and
   ^.*; between ^.* and PIPE; before 'xcd_run HRS'.

   NOTE: any time you modify ~ldm/etc/pqact.conf, you should check your
   editing for errors by running (as the user 'ldm'):

   ldmadmin pqactcheck
   
4) the LDM pattern-action file, ~ldm/etc/pqact.conf, MUST be modified to
   include entries to run ldm-mcidas decoders.  The decoders of interest
   will be pnga2area (converts PNG compressed imagery in AREA format
   to McIDAS AREA files), pngg2gini (converts PNG compressed imagery in
   GINI format to GINI files), proftomd (converts FSL wind profiler data
   in netCDF format to McIDAS MD files), and nldn2md (converts NLDN lightning
   data in its broadcast format to McIDAS MD files).

   Example pqact.conf entries for ldm-mcidas decoders can be found in
   the file ldm-mcidas-pqact.conf that is included in the ldm-mcidas
   distribution.  This file can also be found on the web at:

   http://www.unidata.ucar.edu/packages/mcidas/mcidd/ldm-mcidas-pqact.conf
   

The configuration of the McIDAS-XCD decoders is handled in the McIDAS
environment (i.e., as the user 'mcidas').  The process for setting up
McIDAS-XCD decoding is described in:

Build, Install, and Configure -XCD
http://www.unidata.ucar.edu/packages/mcidas/2002/mcx/mcidas-xcd.html 

The process that this user goes through -- and this should be done
while the LDM is not running -- is basically:

1) login as 'mcidas'

2) cd workdata

3) define the directory in which you want McIDAS-XCD to create output
   data files.  This should be on a file system that has ample
   room, especially if your are going to run the McIDAS-XCD GRIB decoder.
   I typically use a directory named /data/ldm/mcidas.  I setup a link
   in ~ldm so that ~ldm/data/mcidas is /data/ldm/mcidas.  I make sure
   that this directory is readable, writable, and executable by both
   the user 'mcidas' and the user 'ldm'.  These users should be in the
   same group, so the setup is easy.

   Assuming that you have also chosen /data/ldm/mcidas as the output
   directory for McIDAS-XCD decoders, and assuming that this directory
   exists and is writable by the user McIDAS, you then define a McIDAS
   string to point to this directory:

   te.k XCDDATA \"/data/ldm/mcidas

4) define file REDIRECTions to locate McIDAS data files.  This step
   uses the decision you made in step 3):

   cd ~mcidas/data
   cp EXAMPLE.NAM LOCAL.NAM

   <edit LOCAL.NAM and change directory locations for each of the
   data file name masks to match the directory location you decided
   on in step 3)>

   after you have finished editing LOCAL.NAM, make the file REDIRECTions
   you defined in it active:

   cd ~/workdata
   redirect.k REST LOCAL.NAM

5) setup McIDAS XCD decoding:

   cd ~/workdata
   batch.k XCD.BAT
   batch.k XCDDEC.BAT

6) at this point, if you are sure you want to decoded model data (GRIB
   data) into McIDAS GRID files, turn on the McIDAS GRIB decoder:

   decinfo.k SET DMGRID ACTIVE


At this point, you should be able to turn on the LDM and see your system
ingest data and have McIDAS-XCD decode that data.

The next step is defining McIDAS ADDE datasets using the products that will
ingested by the LDM and files that will be created from those products by
ldm-mcidas and McIDAS-XCD decoders.  The process is basically:

1) login as 'mcidas'

2) make a local copy of DSSERVE.BAT and edit it to match your system setup:

   cd ~mcidas/data
   cp DSSERVE.BAT LSSERVE.BAT

   <edit LSSERVE.BAT and modify directory paths to match how you setup
   the LDM to decode and file NNEXRAD and FNEXRAD data>:

   if you are going to be ingesting FNEXRAD composite radar images, then
   you need to modify -- if necessary -- the DIRFILE path listed for
   the FNEXRAD products.  These would be the lines:

DSSERVE ADD NEXRCOMP/1KN0R-NAT  GINI TYPE=IMAGE DIRFILE=/data/ldm/gempak/nport/R
ADAR/1km/n0r/n0r_* "1 km N0R US Base Reflectivity Composite
DSSERVE ADD NEXRCOMP/2KN1P-NAT  GINI TYPE=IMAGE DIRFILE=/data/ldm/gempak/nport/R
ADAR/2km/n1p/n1p_* "2 km N1P US 1-hr Precip. Composite
DSSERVE ADD NEXRCOMP/4KNTP-NAT  GINI TYPE=IMAGE DIRFILE=/data/ldm/gempak/nport/R
ADAR/4km/ntp/ntp_* "4 km NTP US Storm Total Precip. Composite

and 

DSSERVE ADD NEXRCOMP/6KN0R-NAT  AREA TYPE=IMAGE DIRFILE=/data/ldm/gempak/nexrad/
NEXRCOMP/6KN0R-NAT/BREF_*  "6 km N0R National Composite Radar
DSSERVE ADD NEXRCOMP/10KRCM-NAT AREA TYPE=IMAGE DIRFILE=/data/ldm/gempak/nexrad/
NEXRCOMP/10KRCM-NAT/BREF_* "10 km RCM National Composite Radar

   if you are NOT going to be ingesting FNEXRAD composite radar imagery,
   comment these lines out.  A comment in a McIDAS BATCH file starts with
   'REM '.

   if you are going to ingest NEXRAD data from WSI (not likely given that
   you can get the same data for free in the IDD), you will need to uncomment
   lines that define the WNEXRAD datasets).  Again, it is NOT likely that
   you would be doing this

3) after you finish modifying LSSERVE.BAT, make the dataset definitions in
   it active:

   cd ~/workdata
   batch.k LSSERVE.BAT


The key point in all of the above is that you have to:

o decide where to store data products (for both the LDM setup and XCD setup)

o "teach" McIDAS where those data products will be found (the file REDIRECTion
  step)

o configure the LDM to ingest and decode data

o configure McIDAS to decode data

o define ADDE datasets to access the data files that are created by decoders

Lastly, since there will be a number of systems accessing the data through
McIDAS ADDE, you need to setup the McIDAS ADDE remote server on the machine
running the LDM.  This is described in:

Prepare the mcidas and mcadde Accounts
http://www.unidata.ucar.edu/packages/mcidas/2002/mcx/prepare_mcx.html

Basically you will be:

1) create the file ~mcidas/.mcenv using the information in the above
   web page

2) create the account named 'mcadde'.  Make the HOME directory for this
   user the same as for the user 'mcidas'.  Put 'mcadde' into the same
   group as 'mcidas' and 'ldm'.  Setup the 'mcadde' account so that it
   is NOT a login account (default shell will /bin/false)

3) as 'root' run the ADDE setup shell script mcinet2002.sh:

   <login as 'root'>
   cd ~mcidas
   ./mcinet2002.sh install mcadde

   The setup procedure in mcinet2002.sh _assumes_ that you are not using
   NIS/NIS+.

4) if there is a firewall running on this machine (or between this machine
   and others on your network), you will need to configure that firewall
   to allow access through ports 500 and 503.

5) define ADDE access to the datasets you setup previously:

   <login as 'mcidas'>
   cd ~mcidas/data
   cp DATALOC.BAT LOCDATA.BAT

   <edit LOCDATA.BAT and replace the phrase 'fully_qualified_host_identifier'
   with the name of the server that users should go to when wanting to access
   data through ADDE>

   Now, there are several datasets defined in LOCDATA.BAT that you will
   most likely not have on your local machine (e.g., GINICOMP, GINIEAST,
   GINIWEST).  Given this, you would replace 'fully_qualified_host_identifier'
   with a machine name other than your local machine.  Assuming that
   the name of the machine you are going to run the ADDE remote server
   on is atm1.ucdavis.edu (substitute the appropriate machine name for
   this fictitious one), I would suggest defining LOCDATA.BAT entries as
   follows:

DATALOC ADD CIMSS     atm1.ucdavis.edu
DATALOC ADD GINICOMP  pscwx.plymouth.edu
DATALOC ADD GINIEAST  stratus.al.noaa.gov
DATALOC ADD GINIWEST  papagayo.unl.edu
DATALOC ADD NEXRCOMP  atm1.ucdavis.edu
DATALOC ADD RTGRIDS   atm1.ucdavis.edu
DATALOC ADD RTIMAGES  atm1.ucdavis.edu
DATALOC ADD RTNEXRAD  atm1.ucdavis.edu
DATALOC ADD RTPTSRC   atm1.ucdavis.edu
DATALOC ADD RTWXTEXT  atm1.ucdavis.edu
DATALOC ADD TOPO      atm1.ucdavis.edu

   If you are not going to be ingesting and filing the NEXRAD radar composite
   images, you will want to change the NEXRCOMP dataset entry above to
   point at a machine that has the data:

DATALOC ADD NEXRCOMP  papagayo.unl.edu

Lastly, after all of the above is running, you need to setup data file
scouring.  THIS IS IMPORTANT!!!!  Here is the quick and dirty:

1) copy the file mcscour.sh from the ~mcidas/mcidas2002/data/mcscour.sh
   to a directory in the PATH of the user running the LDM.

   Edit the just copied version of mcscour.sh and set McIDAS environment
   variables in the same way that you did for xcd_run.

   Set entries in mcscour.sh to scour back to the number of days of
   McIDAS MD and GRID data that you want to keep.  mcscour.sh is sent
   out to keep only 2 days of MD data and 1 day of GRID data (the
   GRID data is extremely voluminous):

qrtmdg.k GRID 5001 6400 1
doqtl.k  1  70 2
doqtl.k 71  80 2
doqtl.k 81  90 2
doqtl.k 91 100 2
delwxt.k 1 10

   The number of days of MD data to keep is set by the last argument on
   the doqtl.k lines.  The number of days of GRID data is set by the
   last argument on the qrtmdg.k line.  The delwxt.k line should be
   OK as it is sent out.  This line scours text files created by McIDAS-XCD
   decoders.

2) if/when you decide to ingest radar composite images (from the FNEXRAD
   feed) or NEXRAD images from the NNEXRAD feed, please review the
   discussion of data file scouring in the McIDAS web pages and ask
   us questions if you need help.


There is likely to be some tweeks you will want/need to make to all of the
above, but this should basically get you up and working.

Tom Yoksas