20050120: Unidata McIDAS-XCD Trouble

>From: "James T. Brown" <address@hidden>
>Organization: Michigan State University
>Keywords: 200501201933.j0KJXOv2020815 McIDAS-XCD

Hi Jim,

>I am trying to get the XCD decoders to work properly, but not having
>much luck with the MDXX files. The decoders seem to be running and MDXX
>files are being created, but it doesn't appear that the data is being
>written correctly.  When trying to plot data using McGUI, little to no
>data is ever displayed.

Did you create ADDE datasets from the data that is being produced?  In
a nutshell, the pieces of getting McIDAS to access data created by XCD
are (I list the steps just to make sure that you didn't leave out any
step):

- register as a Unidata user

- download the distribution into the 'mcidas' account

- set the McINST_ROOT enviornment variable (typically /home/mcidas if
  this is the HOME directory of the user 'mcidas')

- unpack the distribution

- CD to the McIDAS source directory

  cd mcidas2004/src

- build the distribution:

  make all                   <- builds McIDAS-X and McIDAS-XCD

- install the distribution:

  make install.all

- setup file REDIRECTions that tell McIDAS where to read/write data
  files:

<as 'mcidas'>

cd ~mcidas/data
cp EXAMPLE.NAM LOCAL.NAM
<edit LOCAL.NAM and set the directories to match your setup>

cd ~mcidas/workdata
redirect.k REST LOCAL.NAM

- copy files from the McIDAS distribution to the directory in which
  you want to run XCD and ldm-mcidas decoders.  Example assuming that
  the output data directory is /data/ldm/mcidas:

<as 'mcidas'>

cd ~mcidas/workdata
te.k XCDDATA \"/data/ldm/mcidas

cp ~mcidas/data/SCHEMA /data/ldm/mcidas
cp ~mcidas/data/SYSKEY.TAB /data/ldm/mcidas
cp ~mcidas/workdata/ROUTE.SYS /data/ldm/mcidas

cd /data/ldm/mcidas
chmod 664 SCHEMA SYSKEY.TAB ROUTE.SYS

mkdir bufr grib

- copy the XCD run script (xcd_run) to a directory in the PATH of the
  user running the LDM (typically ~ldm/util or ~ldm/decoders)

- edit the XCD run script and set variables to match your setup (may
  require no modification)

- configure the LDM ~ldm/etc/ldmd.conf file to run the XCD run script:

  exec  "xcd_run MONITOR"

- configure the LDM ~ldm/etc/pqact.conf file to run the XCD decoders
  through the XCD run script:

IDS|DDPLUS      .*      PIPE
        xcd_run DDS
HDS     .*      PIPE
        xcd_run HDS

- setup data scouring for McIDAS-XCD produced data files: run the mcscour.sh
  script from cron for the user running the LDM; copy mcscour.sh from
  ~mcidas/workdata to a directory in the PATH of the user running the LDM.

- run the XCD setup BATCH scripts as 'mcidas'

<as 'mcidas'>

cd ~mcidas/workdata
batch.k XCD.BAT
batch.k XCDDEC.BAT

- optionally enable decoding of GRIB data into McIDAS GRID format (warning:
  this takes a lot of disk space):

decinfo.k SET DMGRID ACTIVE

- stop and restart the LDM

You should be at this point.  You are not, however, finished yet:

- create ADDE datasets for the data that will be decoded by XCD and
  ldm-mcidas decoders:

<as 'mcidas'>

cd ~mcidas/data
cp DSSERVE.BAT LSSERVE.BAT
cp DATALOC.BAT LOCDATA.BAT

edit LSSERVE.BAT and change anything you need to change; NOTE that you
may need to make modifications in LSSERVE.BAT to match where you setup
ldm-mcidas decoders to create output files!  Also, if you are ingesting
and saving NEXRAD Level III radar data, you should change the lines
in LSSERVE.BAT that reference a NEXRAD.CFG file:

change NNEXRAD.CFG to MSUNEXR.CFG

cd ~mcidas/workdata
cp NNEXRAD.CFG MSUNEXR.CFG
<edit MSUNEXR.CFG to set DIRMASK and FILEMASK to regular expressions that
match the directory(ies) and file names for your NEXRAD Level III data files)

batch.k LSSERVE.BAT           <- *** this creates McIDAS ADDE datasets

- setup the remote ADDE server (for access by non-'mcidas' users, this
  is the easiest way to configure for multiple users)

>I built the software on a Sun-Blade 1000 running Solaris 9 (SPARC). The
>LDM is running also running on a Sun-Blade 1000 with Solaris 9 (SPARC)
>- the McIDAS software is accessed via NFS (The ADDE server and LDM are
>on two separate machines, but the McIDAS data directory is on the same
>machine as the LDM).

Since the data is being decoded, it is most likely that the only
thing left to be done is the creation of ADDE datasets.  See above.

>I am not real sure where to begin...
>
>1) XCD Processes Are Running:
>
>% ps -ef | grep inge | grep -v grep
>ldm 12917 12889 5 13:39:33 ? 0:38 ingetext.k DDS
>ldm 12889 12878 0 13:39:31 ? 0:00 ingetext.k DDS
>ldm 12888 12878 0 13:39:31 ? 0:00 ingebin.k HRS
>ldm 12916 12888 1 13:39:33 ? 0:03 ingebin.k HRS
>
>% ps -ef | grep start | grep -v grep
>ldm 12879 12876 0 13:39:31 ? 0:00 startxcd.k
>ldm 12918 12879 0 13:39:33 ? 0:00 startxcd.k

Looks good.

>2) Sample Contents of Data Folder (/data/mcidas):
>
>% date
>Thu Jan 20 13:59:30 EST 2005
>
>% ls -lm MD*
>-rw-rw-r-- 1 ldm wx 49999236 Jan 20 13:39 MDXX0009
>-rw-rw-r-- 1 ldm wx 338536 Jan 20 06:30 MDXX0010
>-rw-rw-r-- 1 ldm wx 44560 Jan 19 18:49 MDXX0019
>-rw-rw-r-- 1 ldm wx 151312 Jan 20 02:53 MDXX0020
>-rw-rw-r-- 1 ldm wx 16640 Jan 19 18:49 MDXX0029
>-rw-rw-r-- 1 ldm wx 112752 Jan 20 13:38 MDXX0030
>-rw-rw-r-- 1 ldm wx 8007456 Jan 20 13:59 MDXX0059
>-rw-rw-r-- 1 ldm wx 887448 Jan 20 06:51 MDXX0060
>-rw-rw-r-- 1 ldm wx 5951016 Jan 20 01:45 MDXX0069
>-rw-rw-r-- 1 ldm wx 17360 Jan 20 13:37 MDXX0070

Looks OK, but some of the data files look awfully small.

>% ls -lm *XCD
>-rw-rw-r-- 1 ldm wx 17418960 Jan 19 18:59 DD050190.XCD
>-rw-rw-r-- 1 ldm wx 2147483647 Jan 20 13:59 DD050200.XCD
>
>% ls -lm ZZ*IDX
>-rw-rw-r-- 1 ldm wx 811008 Jan 19 18:59 ZZ05019.IDX
>-rw-rw-r-- 1 ldm wx 396951552 Jan 20 13:59 ZZ05020.IDX
>
>% ls -lm SCHEMA
>-rw-rw-r-- 1 ldm wx 465408 Jan 19 18:10 SCHEMA
>
>% ls -lm SYSKEY.TAB
>-rw-rw-r-- 1 ldm wx 24000 Jan 20 02:53 SYSKEY.TAB
>
>There are a number of *IDX files and time stamps on many of those files
>indicate that data is being captured.

Looks OK.

>3) PTLIST Output:
>
>% ptlist.k RTPTSRC/PTSRCS FORM=FILE ALL
>Pos Description Schema NRows NCols Proj# Created DataDate
>------ -------------------------------- ------ ----- ----- ----- ------- 
>--------
>9 SAO/METAR data for 19 JAN 2005 ISFC 72 7000 0 2005019 2005019
>10 SAO/METAR data for 20 JAN 2005 ISFC 72 7000 0 2005020 2005020
>19 Mand. Level RAOB for 19 JAN 2005 IRAB 8 1500 0 2005019 2005019
>20 Mand. Level RAOB for 20 JAN 2005 IRAB 8 1500 0 2005020 2005020
>29 Sig. Level RAOB for 19 JAN 2005 IRSG 16 6000 0 2005019 2005019
>30 Sig. Level RAOB for 20 JAN 2005 IRSG 16 6000 0 2005020 2005020
>59 SYNOPTIC data for 19 JAN 2005 SYN 8 6500 0 2005019 2005019
>60 SYNOPTIC data for 20 JAN 2005 SYN 8 6500 0 2005020 2005020
>69 PIREP/AIREP data for 19 JAN 2005 PIRP 24 1500 0 2005019 2005019
>70 PIREP/AIREP data for 20 JAN 2005 PIRP 24 1500 0 2005020 2005020
>ptlist.k: Done

Hmm...  So, it looks like you _do_ have ADDE datasets setup.  Since
you can list the datasets using PTLIST, you should be able to view
their contents through the MCGUI.

>4) XCD Status:
>
>When running "statdisp", the "DDS" and "HRS" data source entries are
>listed in white letters and continually update. The decoders listed in
>the bottom section of the sreen are all listed in red letters.
>
>Here is recent output from "stat.k":
>
>% stat.k
>Decoder Status Display 2005020 190404
>
>## Src Ingestor Time Byte Index Index file Origin Wmo Product
>-- ---- -------- ------ ----------- ------- ----------- ------ ---- -------
>1 DDS INGETEXT 190358 60604080 99392864 ZZ05020.IDX Â?Â?Â?Â? Â?Â?Â?Â?
>4 HRS INGEBIN 190358 7308507 HRS.SPL
>
>GridF/ Grid
>## Decoder Time Begptr Lasptr MD Row Col Text Index
>-- -------- ------ ------- ------- ------ ---- ----- ------------ -------
>1 SAODEC *234623 15056 202720 9 ZZ05019
>2 RABDEC *234623 15072 202720 ZZ05019
>3 SYNDEC *234623 14944 202720 59 8 ZZ05019
>5 *185333 9882835298828352 ZZ05020
>7 PIRDEC *234623 15232 202720 ZZ05019
>14 *185333 9882835298828352 ZZ05020
>15 *185333 9882835298828352 ZZ05020
>stat.k: Done

Moduolo the non-printable characters, this looks OK.

>The information also seems to indicate that the MDXX files are not
>being updated properly. especially the MDXX files that should be
>created by the SAODEC decoder. There are no obvious errors being
>reported in the "ldmd.log" and "xcd_start.log" files.

The PTLIST listing above shows that it can access the data through
correctly defined ADDE datasets.  What does a listing of data look
like:

PTLIST RTPTSRC/SFCHOURLY

>I have built the Unidata-XCD decoders twice and still haven't corrected
>the problem.  I also have the LDM-McIDAS decoders installed (pnga2area)
>and those seem to work just fine. The McIDAS packages also seems to run
>properly - I am only having trouble with the XCD decoders.

I don't think it is a decode problem.

>Do you have any suggestions on what I may try to troubleshoot? Are
>there any other decoders that can be used to create the MDXX files that
>are referred to by the RTPTSRC group?

Run the PTLIST command and let me know the results.

>We are hoping to use the MDXX files and MCGUI for instructional
>purposes. If I am unable to get the XCD decoders to work properly, I
>will reconfigure the McGUI to point to a remote ADDE server, but since
>I have access to IDD, it would be nice to setup a local ADDE server for
>our instructional lab and users throughout my Department.

I agree.

>I am sure I am overlooking some simple step, but seem to be stuck at
>the moment.  Again, data seems to be coming in, but I am not sure the
>MDXX files are being written to properly. If they are, I am unable to
>access using McGUI. This doesn't appear to be a McGUI problem because I
>am able to plot if I point to a remote ADDE server.

Can I get on your system as 'mcidas' to take a quick look?

>By the way, speaking of remote ADDE servers - There seems to be a list
>of sites available in the "Configure | ADDE Dataset Location" menu of
>the McGUI application. Is permission recommended before accessing
>remote ADDE servers?

No.  These are servers that owners have allowed general access to
for the Unidata community.

>I have a faculty member that would like to access
>data from the RTPTSRC group next week and if I am unable to get things
>to work locally, I may need to have her point to a remote ADDE server.

It really does sound like you setup everything correctly.  I guess
that the quickest thing would be for me to login and look around
to see what I can see.  If you setup the ADDE remote server for your
machine, I could try out access to the data from here; this may point
out some problems, and, then again, it might not.

>Thanks.

No worries.

Cheers,

Tom
--
NOTE: All email exchanges with Unidata User Support are recorded in the
Unidata inquiry tracking system and then made publicly available
through the web.  If you do not want to have your interactions made
available in this way, you must let us know in each email you send to us.