20000303: McIDAS-XCD setup and LDM quesitons at Colgate (cont.)

>From: Adam Burnett <address@hidden>
>Organization: Colgate
>Keywords: 200002291726.KAA26578 McIDAS-X 7.60

Adam,

>Thanks for the help in building Mcidas 7.6.  Now I have another problem and
>it seems unusually weird.  It involves my efforts to get the ldm working
>with mcidas-xcd.  I had most of this working with my 7.4 version but for
>some reason I have these new issues.  The basic problem is that I'm not
>getting my MD files and grid files decoding from xcd.  In addition, I'm
>having strange behavior with my ldm (which may explain my MD and GRID
>problems). Let me start with the LDM.

Here goes...

>When I upgraded to Mcidas 7.6, I used the new copy of xcd_run to replace my
>old copy.  I probably didn't need to do this but I wanted to make sure that
>everything was up to date.

This was not needed, but it should cause no problems as long as the
definitions for MCDATA, MCPATH, etc. are set the same as they were in the
old copy of xcd_run.

>I then made the necessary changes to xcd_run.
>My mcidas installation is in /data2/mcidas. I want xcd to decode data into
>/data3/xcd  (for your information, my area files are going into
>/data3/mcfiles)

OK.

>My xcd_run looks like:
>
>MCHOME=/data2/mcidas
>MCDATA=/data3/xcd

MCDATA should be:

MCDATA=/data2/mcidas/workdata

It is the McIDAS working directory for the user 'mcidas'.

>MCLOG=$MCDATA/XCD_START.LOG
>MCPATH=${MCDATA}:$MCHOME/data:$MCHOME/workdata:$MCHOME/help

MCPATH should be:

MCPATH=${MDATA}:$MCHOME/data:$MCHOME/help

># Setup PATH so that the McIDAS-X executables can be found
>
>PATH=$MCHOME/bin:/usr/openwin/bin:/usr/openwin/share/include:/bin:/data1/pro
>gram/SUNWspro/bin:/usr/ccs/bin:/usr/bin:/usr/local/bin:/usr/ucb

This looks correct

># Set LD_LIBRARY_PATH to include all directories (other than those searched
># by default) that are needed to be searched to find shared libraries.
># For this example, I assume that the shared Fortran library is located
># in /opt/SUNWspro/SC4.2/lib
>
>LD_LIBRARY_PATH=/usr/openwin/lib:/data1/program/SUNWspro/SC4.2/lib:/data1/pr
>ogram/SUNWspro/lib:$MCHOME/lib:/usr/lib:/usr/ucblib

OK.  If the Sun compilers are installed correctly, this could be set to
be:

LD_LIBRARY_PATH=/usr/openwin/lib:/data1/program/SUNWspro/lib:$MCHOME/lib:/usr/lib:/usr/ucblib

>My pqact.conf file has the following lines:
>
>#Entries for Mcidas-XCD decoders
>#
>DDPLUS|IDS     ^.*     PIPE    xcd_run DDS
>HRS    ^.*     PIPE    xcd_run HRS
>#

Assuming tabs in appropriate places, this should be fine.  I personally
prefer to put the portions of the invocation on separate lines:

DDPLUS|IDS      ^.*     PIPE
        xcd_run DDS
HRS    ^.*      PIPE
        xcd_run HRS

but this is not absolutely necessary.

>My ldmd.conf file has these lines.  I've always had the pqsurf commented out
>but I don't know why.
>
>#
>exec   "pqexpire"
>exec   "xcd_run MONITOR"
>exec   "pqbinstats -d /data2/ldm/logs"
>exec   "pqact -d /data2/ldm /data2/ldm/etc/pqact.conf"
># exec "pqsurf"

This looks good.  It assumes that xcd_run is in the PATH of the user 'ldm'.

>After I made sure that I had things set correctly, I ran the ldmadmin start
>command.  Normally, the program returns a message saying that ldm has
>started.  Now it just sits there:
>
>gissun% ldmadmin start
>starting the LDM server...

At this point I would kill ldmadmin and kill all LDM processes.  That
way you can start fresh.

>I open another command window and run the ldmadmin watch command, and it
>appears as if stuff is comming in.  Here is an example:
>
>gissun% ldmadmin watch
>(Type ^D or ^C when finished)
>Mar 03 15:08:45 pqutil:     1651 20000303141541.053 IDS|DDPLUS 149  ASUS45
>KGTF 031415 /pSWRSK
>Mar 03 15:08:45 pqutil:      759 20000303141546.599 IDS|DDPLUS 155  SAUS70
>KWBC 031413 /pMETAR
>Mar 03 15:08:45 pqutil:      334 20000303141546.780 IDS|DDPLUS 156  FTXX62
>KWBC 031411
>Mar 03 15:08:45 pqutil:      104 20000303141546.783 IDS|DDPLUS 157  SAFR32
>LFPW 031400 CCA
>Mar 03 15:08:45 pqutil:      256 20000303141546.787 IDS|DDPLUS 158  UDEU40
>ESWI 031408
>Mar 03 15:08:45 pqutil:      781 20000303141546.796 IDS|DDPLUS 160  FPAK11
>PAFA 031411

The hanging of ldmadmin (which is simply a Perl script that, in this
mode ('start') is looking for indication that processes have started)
usually is an indication that your OS is lacking some needed pathes.

>Also, my mcidas decoders for area, nids, and lightning are working
>perfectly.

OK, the LDM is running at this point, but ldmadmin has not been able
to determine that.  I will have to check with our system administrator
to find out if there is a specific OS patch that needs to be applied,
or if a series of patches is needed.

>The XCD-START.LOG looks like:
>
>Starting DDS at 00063.150757
>Domestic Data Service
>150757
>High Resolution Data Service

Looks OK.

>When I go to stop ldm with the ldmadmin stop command I get the following
>message:
>
>stopping the LDM server...
>Mar 3 15:02:59 UTC gissun.colgate.edu : make_lockfile: another ldmadmin
>process exists
>gissun%

This is related to the same problem as ldmadmin never returning with an
indication that the LDM has been started.

>Meanwhile, I am getting tons and tons of stuff dumped into my /data3/xcd
>directory.  I'm not even sure what most of this stuff is. What is all this
>stuff? (this is one of the burning questions I've had for about 2 years).

The majority of things will be files with the .IDX suffix.  These
are index files by product type into the daily spool file that has the
.XCD suffix.  The index files are needed for text access to the data
files.

>I
>don't seem to be getting any MD or GRID files written to the /data3/xcd
>directory. 

I am a little suspicious of your setup given the settings of MCDATA and
MCPATH in your xcd_run.  The way I found that it was best to run
XCD was to set MCDATA and MCPATH the way I indicated and have file
REDIRECTions that point to where you want various files read/written.
This was the purpose of the settings in EXAMPLE.NAM which you are advised
to copy to LOCAL.NAM and edit to match your system setup.

>One other piece of infomation - I don't remember if I should be seeing
>xcd_run "things" happening in my ldm log.

No, not really.  xcd_run is a shell script wrapper around the XCD
routines startxcd.k, ingebin.k, and ingetext.k.  These routines know
nothing about the LDM, so you would not expect to see logging information
sent to the LDM log.

>All I see are nids, area, and lightning stuff.

OK, so the ldm-mcidas setup in pqact.conf is correct.

>Sorry to present this mystery.  Is it too weird?

No, it is not too weird.  What I would do is the following:

<login as 'ldm'>
o stop the LDM
o make sure that all LDM processes have ended; if they have not, you may
  need to kill them by hand
o after all of the LDM processes have exited, then run 'ldmadmin stop'
  once more.  This time it may complain, but it should return you back
  to the OS command prompt

<login as 'mcidas'>
o to start with a clean slate, cd to the /data3/xcd directory and remove
  all of the .IDX, .XCD, .RAP, .RAT files.  A real cleanup should leave
  you with the files SCHEMA and SYSKEY.TAB in this directory.  SYSKEY.TAB
  should be a link to the copy in the directory in which you are storing
  AREA files.  SCHEMA should be the one from the 7.6 distribution and
  it should also be a link to the copy in the directory where ldm-mcidas
  point source decoders (nldn2md and proftomd) are creating their
  output
o after cleaning out the XCD data directory, then start going through
  the McIDAS installation and configuration instructions in our
  online documentation.  A quick overview of the things you will be
  doing is:

  cd ~mcidas/data
  cp EXAMPLE.NAM LOCAL.NAM
  <edit LOCAL.NAM and set directories to match your system setup>
  cp DSSERVE.BAT LSSERVE.BAT
  cp DATALOC.BAT LOCDATA.BAT
  <edit LOCDATA.BAT and change "fully_qualified..." to the full name
  of your machine)
  cd ~mcidas/workdata
  <start a McIDAS-X session>
  BATCH XCD.BAT
  REDIRECT REST LOCAL.NAM
  TE XCDDATA "/data3/xcd
  BATCH XCDDEC.BAT
  BATCH LOCDATA.BAT
  BATCH LSSERVE.BAT

At this point, your system should be ready for XCD to run correctly.

<login as 'ldm'>
ldmadmin start

Let me know what happens.  By the way, did you do the ADDE remote server
installation as pre the web pages?  If not, you should, but you will
need 'root' access to do it.

>Thanks 

If things still don't work, I will be happy to login and look around.
If you want me to do that, I will need the login and full machine
name.

Talk to you later...

Tom