20050301: ldm feed / mcidas problem

>From: unidata <address@hidden>
>Organization: NMSU/NSBF
>Keywords: 200503011553.j21FrBv2001180 McIDAS-XCD LDM

Hi John,

The best place to send inquiries like this are Unidata User Support
<address@hidden>.

>I just did a clean install of McIDAS on a system here and I am getting
>continuous errors in both the XCD_START.LOG and ldmd.log that appear to
>be related.
>
>The ldmd.log  reports: 
>Mar 01 15:03:46 psnldm pqact[18433]: pbuf_flush (5) write: Broken pipe
>Mar 01 15:03:46 psnldm pqact[18433]: pipe_put: xcd_runDDS write error
>Mar 01 15:03:46 psnldm pqact[18433]: pipe_prodput: trying again
>Mar 01 15:03:46 psnldm pqact[18433]: pbuf_flush (5) write: Broken pipe
>Mar 01 15:03:46 psnldm pqact[18433]: pipe_put: xcd_runDDS write error
>Mar 01 15:03:46 psnldm pqact[18433]: pbuf_flush (3) write: Broken pipe

This can be caused by a number of different things:

- the Bourne shell script 'xcd_run' can not be located through the PATH
  you had in effect when you started your LDM

- there is some sort of error in 'xcd_run'.  This would be in the
  definition of MCHOME, MCDATA, MCPATH, PATH, or LD_LIBRARY_PATH
  inside of xcd_run

- 'xcd_run' executed from pqact.conf entries pipes data (either HRS or DDS)
  to a McIDAS-XCD ingestion routine (ingebin.k for HRS data and ingetext.k
  for DDS data).  If those processes can not write to output spool files,
  they will fail and you will get errors like the ones you show above.

>and the XCD_START.LOG reports:
>Starting HRS at 05060.150610
>problem with file CIRCUIT.DAT
>may need to be initialized with circuit command
>ingebin.k: Done
>Starting DDS at 05060.150610
>problem with file CIRCUIT.DAT
>may need to be initialized with circuit command
>ingetext.k: Done

This looks like the necessary configuration steps were not performed or
that there was a failure at some point.  You should find a file named
CIRCUIT.DAT in the ~mcidas/workdata directory.  For a new installation,
this file will have a new timestamp.  For an upgrade, it may have an
old timestamp.  It is created when the user 'mcidas' runs the
XCDDEC.BAT script as part of the XCD setup phase (which must be done
before attempting to process data ingested by the LDM).

>I ran mcidas and did a CIRCUIT LIST and found everything was inactive, so I
>SET the circuits  to active, listed them and verified they were active.

This is _not_ what you should do.  The CIRCUIT.DAT file is used when
XCD is reading from data being ingested through serial ports.  Folks
using the LDM to get data are not accessing serial ports

>But the logs still keep giving the error messages. 

>Any suggestions as to what I missed in the setup would be appreciated. 
>(I am running this in RH9 Linux on a X86 machine)

Here is a quick reminder for the steps to be followed by 'mcidas' (forgive
the over statement of the list, but I want to be complete):

1) register as a Unidata member through our web portal

2) have the system administrator create a 'mcidas' account that has
   sufficient disk space to download the package; do the build; install
   the resultant binaries and data files; etc.

   While at it, the system administrator should create an account
   named 'mcadde'.  This account should:

   - have the same HOME directory as 'mcidas'
   - _not_ be a login account
   - be in the same group as 'mcidas' and 'ldm'

3) download the McIDAS distribution files: mcidasx2004.tar.Z, mcinstall
   and mcinet2004.sh

4) define the Unix environment variable McINST_ROOT (if it is not already
   defined). Here is the standard definition:

McINST_ROOT=/home/mcidas

5) make 'mcinstall' executable and unpack the distribution:

<as 'mcidas'>
cd ~mcidas
./mcinstall

6) define the envionment variables needed to build the various packages
   include with McIDAS (e.g., netCDF, zlib, HDF, etc.).  Here are
   the settings for Linux:

CC=gcc
CPPFLAGS="-DNDEBUG -Df2cFortran"
CFLAGS=-O
FC=g77
FFLAGS "-O -Wuninitialized -fno-automatic"
CXX=" "
F90=" "

    It is easiest to define these in your shell-specific configuration
    file (e.g., .chsrc for C shell users; .bash_profile for BASH users;
    etc.)

7) build and install the McIDAS-X and -XCD components of the distribution:

<as 'mcidas'>
cd mcidas2004/src
make all
make install.all

8) at this point, you are ready to configure for XCD.  First, identify
   a directory where you are going to want XCD routines to decode data.
   A typical location is /data/ldm/mcidas; your location may be different
   depending on how you setup things.

   Create this output directory if it doesn't already exist AND make
   sure that it is readable AND writable by both 'mcidas' and 'ldm'.
   Also, if the directory already exists, make sure that any files
   in it are readable AND writable by both 'mcidas' and 'ldm'.

9) using /data/ldm/mcidas as the example output directory...

   Define the set of file REDIRECTions needed to locate McIDAS files in the
   output directory:

<as 'mcidas'>
cd ~mcidas/data
cp EXAMPLE.NAM LOCAL.NAM
<edit LOCAL.NAM and set the directory portion of each REDIRECTion to
 that directory.  Take care to _not_ change the REDIRECTion entries
 that point to ~mcidas/data or subdirectories therein. 
cd ~mcidas/workdata
redirect.k REST LOCAL.NAM

10) define the McIDAS string XCDDATA to be the name of the XCD output
    directory:

<as 'mcidas'>
cd ~mcidas/workdata
te.k XCDDATA \"/data/ldm/mcidas

11) run the McIDAS BATCH files that create needed output files for
    XCD to work:

<as 'mcidas'>
cd ~mcidas/workdata
batch.k XCD.BAT
batch.k XCDDEC.BAT                  <- this should create CIRCUIT.DAT and
                                       other files needed by XCD processes.
                                       If it doesn't, stop here and let
                                       me know the contents of your 
                                       ~mcidas/workdata directory.  A
                                       long listing would be most useful:

                                       <as 'mcidas'>
                                       id
                                       cd ~mcidas/workdata
                                       ls -alt

At this point, your 'mcidas' account should be setup enough to run XCD
decoders from LDM pqact.conf actions.

Setup LDM configuration files to run XCD decoders

1) copy xcd_run from the McIDAS distribution to a directory in 'ldm's PATH:

<as 'ldm'>
cp ~mcidas/bin/xcd_run ~ldm/decoders
<edit ~ldm/decoders/xcd_run and set the environment variables to match your
installation>

2) copy mcscour.sh from the McIDAS distribution to a directory in 'ldm's PATH:

<as 'ldm'>
cp ~mcidas/workdata/mcscour.sh ~ldm/util
<edit ~ldm/util/mcscour.sh and set the environment variables to match your
installation>

3) add the execution of the xcd_run MONITOR action to your ~ldm/etc/ldmd.conf
   file:

<as 'ldm'>
cd ~ldm/etc
<add the following line to ldmd.conf:

exec    "xcd_run MONITOR"

4) add the lines needed to run xcd_run ingest actions to your
   ~ldm/etc/pqact.conf file:

<as 'ldm'>
cd ~ldm/etc
<add the following lines to pqact.conf:

IDS|DDPLUS      .*      PIPE
        xcd_run DDS
HRS     .*      PIPE
        xcd_run HRS

    NOTE: white space are tabs everywhere except between xcd_run and either
    DDS or HRS!!!!

5) verify that 'ldm' can write to the XCD output directory:

touch /data/ldm/mcidas/xxx

   If this fails, the decoders won't work, so alter the directory permissions
   accordingly.

   If it succeeds, remove the test file:

rm /data/ldm/mcidas/xxx


At this point, you _should_ be able to start your LDM and decode data
using the XCD decoders.  If continue to get the error messages like
the ones you listed above, it means that a directory permission is
incorrect; a setting in the 'xcd_run' script is incorrect; or that 'ldm'
does not have execute permission for McIDAS executables.

Please let me know if the above helped pinpoint the problem you are seeing.

Cheers,

Tom
--
NOTE: All email exchanges with Unidata User Support are recorded in the
Unidata inquiry tracking system and then made publicly available
through the web.  If you do not want to have your interactions made
available in this way, you must let us know in each email you send to us.