20031124: LDM and McIDAS setup at Western Michigan (cont.)

>From: Unidata Support <address@hidden>
>Organization: UCAR/Unidata
>Keywords: 200310251920.h9PJKRgf026027 McIDAS install

Karthik,

I setup the LDM on s400 to ingest data from the University of Illinois
IDD relay node flood.atmos.uiuc.edu.  The following feeds are now being
ingested:

WMO     - global observations and NCEP NOAAPORT model output
FNEXRAD - NEXRAD Level III composite imagery created by Unidata
NNEXRAD - NEXRAD Level III products
FSL2    - FSL wind profiler data
UNIWISC - Unidata-Wisconsin GOES satellite image sectors and products

Yesterday evening I sent a note to the University of Michigan asking
them to allow s400 to feed from their IDD node, yin.sprl.umich.edu.
Today, I sent a note to SUNY Albany asking them to allow you to feed
the NLDN lightning data from their IDD node, gusher.atmos.albany.edu.

As soon as we hear back from U Michigan and SUNY Albany, I will
uncomment appropriate request lines to the LDM configuration file
/cutrim1/ldm/etc/ldmd.conf and then stop and restart the LDM.

You can monitor how well the data is being ingested through the
Unidata real time statistics pages:

Unidata HomePage
http://my.unidata.ucar.edu
  Software
  http://my.unidata.ucar.edu/content/software
    IDD
    http://my.unidata.ucar.edu/content/software/idd
      Current Operational Status
      http://my.unidata.ucar.edu/content/software/idd/iddgeneral.html
        Real Time Statistics
        http://my.unidata.ucar.edu/content/software/idd/rtstats/index.html

From the last link, click on the 'Statistics by Host' link and then
scroll down until you find a link for the Western Michigan LDM,
s400.wood.wmich.edu.  Clicking on that link will take you to a page
that shows each feed that s400 is ingesting.  There are a variety of
links that will take you to plots of things like latency, log(latency),
histograms of latency, data volume, number of products, and topology
(who the data is coming from).  I recommend visiting the pages to get
an idea of how much data is being ingested by s400 and how well it is
keeping up (through the latency plots).

I setup McIDAS-XCD and ldm-mcidas decoders to decode the data
coming in on the datastreams above.  All decoded data is currently
being written to subdirectories of /cutrim1/ldm/data.  If Elen wants
more data ingested (e.g., the NEXRAD Level II data or other model data,
or the NOAAPORT GINI imagery), we may eventually have to use some of
the space on /cutrim2.  We can worry about that if/when the time comes,
however.

Along with the ingest and decode of data into McIDAS-compatible
formats, I setup scouring of the data files and access to them through
the McIDAS ADDE remote server (running from the 'mcadde' account):

- the scouring was done by running various scour scripts from
  'ldm's crontab

- McIDAS ADDE service was setup by configuring the file .mcenv in
  the ~mcadde directory.  Since the ADDE setup script always defaults
  to use of the HOME directory of the user 'mcidas', and since McIDAS-X
  v2003 is _not_ installed in ~mcidas, I did the following:

  - remove the McIDAS installation in ~mcidas.  This was old and 
    unsupported (v7.6), so the removal should have no impact

  - made links for a set of directories that point to the actual
    McIDAS installation point, /cutrim1/mcidas:

cd ~mcidas
ls -alt
 ...
lrwxrwxrwx  1 mcidas  ldm  20 Dec 24 16:38 help -> /cutrim1/mcidas/help
lrwxrwxrwx  1 mcidas  ldm  24 Dec 24 15:35 savedata -> /cutrim1/mcidas/savedata
lrwxrwxrwx  1 mcidas  ldm  19 Dec 24 15:34 inc -> /cutrim1/mcidas/inc
lrwxrwxrwx  1 mcidas  ldm  19 Dec 24 15:34 lib -> /cutrim1/mcidas/lib
lrwxrwxrwx  1 mcidas  ldm  19 Dec 24 15:34 man -> /cutrim1/mcidas/man
lrwxrwxrwx  1 mcidas  ldm  19 Dec 24 15:34 tcl -> /cutrim1/mcidas/tcl
lrwxrwxrwx  1 mcidas  ldm  21 Dec 24 15:34 admin -> /cutrim1/mcidas/admin
lrwxrwxrwx  1 mcidas  ldm  20 Dec 24 15:34 data -> /cutrim1/mcidas/data
lrwxrwxrwx  1 mcidas  ldm  24 Dec 24 15:34 workdata -> /cutrim1/mcidas/workdata
lrwxrwxrwx  1 mcidas  ldm  19 Dec 24 15:33 bin -> /cutrim1/mcidas/bin
 ...

We should consider reinstalling (_not_ copying) McIDAS under ~mcidas.
It would make the McIDAS installation a lot cleaner.  If we do this,
however, it will require some modifications in the McIDAS-related
scripts in /cutrim1/ldm/util (mcscour.sh, batch.k, xcd_run).

As far as I can tell, everything is working OK except LDM logging.
I will have to touch base with Leonard about this when he gets back
from his vacation.  Until we can sort through why syslog logging
is not working correctly, I have done the following:

- modified /cutrim1/ldm/bin/ldmadmin and set the '-l logs/ldmd.log'
  flag on rpc.ldmd

- since the normal log rotation mechanism (ldmadmin newlog) will
  not work now, I setup rotation of the LDM log files using
  newlog:

#
# rotate LDM logs
#
0 18 * * * /cutrim1/ldm/bin/newlog logs/ldmd.log 14

I have tested the McIDAS ADDE access to the data being decoded on
s400 from the 'mcidas' account on s400 and it works correctly.
I don't know if other machines at Western Michigan will be able
to do the same, however.  If they are not, it means that there is
some sort of a firewall setup that is blocking them from accessing
the ADDE remote server on s400.  A problem like this will take
a system/firewall administrator to correct.

That's all for today.

Cheers,

Tom