20010728: ADDE access to NEXRAD Level III data

>From: "Wayne Bresky" <address@hidden>
>Organization: Cornell
>Keywords: 200107261641.f6QGft109510 McIDAS ADDE NEXRAD

Wayne,

>I haven't tackled the syslog problem yet.  I'll do that on Monday.

OK, sounds good.

>In the meantime, I was wondering about the NEXRAD data.  I
>am receiving BGM N0R data as the listing below indicates
>
> ls -l mcidas/nexrad/RADAR/BGM/N0R
>
>-rw-rw-r--    1 ldm      Unidata     17378 Jul 28 21:05 N0R_20010729_0106
>-rw-rw-r--    1 ldm      Unidata     18308 Jul 28 21:15 N0R_20010729_0115
>-rw-rw-r--    1 ldm      Unidata     19071 Jul 28 21:25 N0R_20010729_0125
>-rw-rw-r--    1 ldm      Unidata     19787 Jul 28 21:37 N0R_20010729_0137
>-rw-rw-r--    1 ldm      Unidata     20533 Jul 28 21:47 N0R_20010729_0147
>
>But when I tried to do an IMGLIST I get the error:
>
>Image file directory listing for:RTNEXRAD/N0R
>imglist.k: There are no images that meet the selection criteria
>imglist.k: done
>
>Do I need a REDIRECT entry to point it to the data?

The NEXRAD data is an example of data that can only be accessed in McIDAS
by ADDE.  The reason is that it is not in a standard McIDAS format like
AREA.

Setting up the ADDE access to the data is done in a two step process:

1) making the ADDE definitions in LSSERVE.BAT (in ~mcidas/data; a copy of
  ~mcidas/data/DSSERVE.BAT) active:

   cd ~mcidas/workdata
   te.k XCDDATA \"<whatever directory you chose>
   batch.k LSSERVE.BAT

   (changes to LSSERVE.BAT may be desired before doing this step.  see
   below for information pertinent to the NEXRAD data)

2) editing the file ~mcidas/workdata/NNEXRAD.CFG and setting directory
   and file name information to match how you setup your LDM to file
   the NEXRAD products.

Before jumping into 2), I recommend doing the following:

1) make a copy of NNEXRAD.CFG with a different name so it won't get overwritten
   by a new McIDAS install:

   cd ~mcidas/workdata
   cp NNEXRAD.CFG CORNEXR.CFG

2) edit LSSERVE.BAT and change the INFO= field for the RTNEXRAD dataset
   entries (make sure you get all of the entries) from INFO=NNEXRAD.CFG
   to INFO=CORNEXR.CFG

3) while you editing LSSERVE.BAT, also comment out all datasets that you
   will not (maybe never?) have.  This would most likely be the following
   list:

   WNEXRAD  <- NIDS products from WSI Corporation
   WNOWRAD  <- NOWRAD composite imagery from WSI Corporation
   
   If you are not going to be decoding GRID data into McIDAS format (this
   is turned off by default in the XCD setup), then you can also comment
   out the entries for RTGRIDS.

   IF you contract with WSI for their NOWRAD composite imagery, you will
   want to keep (i.e., not comment out) the WNOWRAD entries.

   You comment out each line for the particular dataset by inserting
   'REM ' at the beginning of that line:

change:

DSSERVE ADD WNEXRAD/BREF1         NEXR    1 9999 TYPE=IMAGE INFO=WNEXRAD.CFG   
"Base Reflectivity Tilt 1

to:

REM DSSERVE ADD WNEXRAD/BREF1     NEXR    1 9999 TYPE=IMAGE INFO=WNEXRAD.CFG   
"Base Reflectivity Tilt 1
 
   and so on.

4) since you have probably already made the entries in LSSERVE.BAT active
   by a previous invocation of 'batch.k LSSERVE.BAT' from ~mcidas/workdata,
   you should delete the previous definitions so you can remake the new
   set and not keep unwanted/unneeded ones from the previous setup run:

   cd ~mcidas/workdata
   rm RESOLV.SRV
   batch.k LSSERVE.BAT

5) finally, it is time to edit CORNEXR.CFG:

   cd ~mcidas/workdata
   <edit CORNEXR.CFG>

   change:

DIRMASK=/data/ldm/nport/nexrad/\ID/\TYPE
FILEMASK=\TYPE_*.\ID

   to match your directory and file name scheme.  Above, you show:

   ls -l mcidas/nexrad/RADAR/BGM/N0R

   the DIRMASK entry in the .CFG file needs to be a fully qualified pathname.
   If mcidas/nexrad/RADAR/BGM/N0R is actually /data/mcidas/nexrad/RADAR/BGM/N0R
   you would make DIRMASK:

DIRMASK=/data/mcidas/nexrad/RADAR/\ID/\TYPE

   The "replaceables" \ID and \TYPE are symbolic representations of the
   station IDs and product types, respectively.  Do not change these.

   You also need to identify how the files are named.  If you are
   naming them the way that is recommended for GEMPAK, they would look
   like (this is what your list shows):

   N0R_CCYYMMDD_HHMM

   In this case, you would set FILEMASK to:

FILEMASK=\TYPE_*

   (there is no replaceable syntax for CCYYMMDD_HHMM (or any part thereof)
   yet).

After making the changes indicated above, you should be able to access
the data from your 'mcidas' account.

Here is where I get back on my soap box about setting up the remote
server so clients always go through that interface (sorry, I have to do
this for those souls who will be reading the archives for information
in the future).

If you don't setup access to one remote server, then each user will
have to go through the process above.  This is a _pain_ and will
undoubtedly lead to problems in the future (for novice users at
least).  Now, if all clients go to the remote server, and if your setup
above was in the 'mcidas' account on the remote server, and if the
remote server is setup correctly, then all clients have to do to access
the data is to do a DATALOC for the RTNEXRAD dataset in their McIDAS
session.  This would look something like:

DATALOC ADD RTNEXRAD fully_qualified_name_of_ADDE_remote_server

>Also, I had a question concerning setting up the ADDE server.  I followed
>the instructions for RH 7.x in the notes, so I assume I do NOT follow the
>ADDE setup in the McIDAS 7.7 install instructions.

Right you are.  I have not had the time to rewrite mcinet7.x.sh to play
by RedHat 7.x rules.  The problem is complicated by the fact that other
Linux bundlers like Debian and Slackware do not use xinetd. They still
use inetd, so the mcinet7.x.sh script is still the way to set them up.
(editorial: Linux is not Linux is not Linux... bummer).

>I assume I do not have to modify /etc/services.  Or do I?

According to the man page for xinetd, _if_ the service(s) is(are)
listed in /etc/services, then you have to be careful to make sure that
the entries in /etc/xinetd.d/mcserv and mccompress match what is in
/etc/services.  As I read the man page, you do not have to put the
serivces in /etc/services.  All the information that xinetd needs is in
mcserv and mccompress:

# mcservsh (port 500)
service mcserv
{
        log_type        = SYSLOG authpriv debug
        flags           = REUSE NODELAY
        socket_type     = stream
        protocol        = tcp
        wait            = no
        port            = 500
        user            = mcadde
        server          = /home/mcidas/bin/mcservsh
        server_args     = -H /home/mcidas
        log_on_success  += USERID HOST DURATION
        log_on_failure  += USERID HOST RECORD
}

# mccompress (port 503)
service mccompress
{
        log_type        = SYSLOG authpriv debug
        flags           = REUSE NODELAY
        socket_type     = stream
        protocol        = tcp
        wait            = no
        port            = 503
        user            = mcadde
        server          = /home/mcidas/bin/mcservsh
        server_args     = -H /home/mcidas
        log_on_success  += USERID HOST DURATION
        log_on_failure  += USERID HOST RECORD
}

You have to remember to send a USR1 signal to xinetd after creating
mcserv and mccompress in /etc/xinetd.d.  You should also make sure that
the file permissions on these files are rw-r-r (644) and are owned by
'root' in group 'root'.

Now, if the remote service doesn't work, and if your ~mcidas/.mcenv
entries are correct (no typos; correct PATH and LD_LIBRARY_PATH;
correct export of the various enviornment variables); and the
~mcidas/mcidas/data directory exists and has permission for the user
'mcadde' to read/write in that directory, then I would add the services
to /etc/services and send a USR1 signal to xinetd:

/etc/services entries for McIDAS ADDE remote server:

mcserv          500/tcp                         # McIDAS ADDE port
mccompress      503/tcp                         # McIDAS ADDE compression port

(mind tabs for white space)

kill -USR1 <pid of xinetd>

What is the name of the machine you are setting this up on?  Will the
McIDAS ADDE ports (500 and 503) be accessible to the outside (outside
of Cornell that is) world?  If it is, I can test out stuff from home
(where I am now) or work.  ADDE is very cool this way!

Tom