20020724: McIDAS-X & XCD (cont.)

>From: "Fingerhut, William A" <address@hidden>
>Organization: Lyndon State
>Keywords: 200207191535.g6JFZa920941 McIDAS-X XCD REDIRECT

Bill,

>Steven knows and approves of your logging in.

OK, I logged on right after reading this message.

I immediately reviewed environment variable settings (e.g., MCPATH,
etc.), REDIRECTions, etc.  All looked OK.

I did all of this -- as always should be done -- from the
~mcidas/workdata.  The key point is when configuring McIDAS, always
run commands while residing in the $MCDATA directory.  This goes for
'mcidas' as well as ordinary users.

While checking out things, I did the following:

<as 'ldm'>
ldmadmin stop

<as 'mcidas'>
cd /var/data/ldm/mcidas/xcd
rm *.R* *.IDT
cd /home/mcidas/workdata
tl.k                     <- to verify XCDDATA setting which was correct
rm ../data/SKEDFILE      <- these should be in ~mcidas/workdata
rm ../data/STRTABLE                       "
rm -f core.*             <- core files from ingetext.k and ingebin.k (!?)

Then, I started looking around.  I found the following:

1) you have your system setup to save XCD-decoded files in 
   /var/data/ldm/mcidas/xcd and ldm-mcidas-decoded files in
   /var/data/ldm/mcidas/xdata.  As far as the software goes, there is
   no reason to keep these sets of files separated from each other.
   In fact, I now recommend to write all data files to a single
   directory as it eliminates a configuration step that I will get
   to next.

   Both the ldm-mcidas and XCD decoders are setup to read/write items
   from the files ROUTE.SYS and SYSKEY.TAB.  For point source file
   decoding, they both use information from the file SCHEMA.  I found
   that you had one copy of SCHEMA in /var/data/ldm/mcidas/xcd and
   a different one (the most current one) in /var/data/ldm/mcidas/xdata.
   Since the copy of SCHEMA that both decoders should be using is
   the most current one, I did the following:

   cd /var/data/ldm/mcidas/xdata
   rm ../xcd/SCHEMA
   ln SCHEMA ../xcd

   Since both decoders want to read/write ROUTE.SYS and SYSKEY.TAB,
   I did the following:

   <still in /var/data/ldm/mcidas/xdata>
   ln ROUTE.SYS ../xcd
   rm ../xcd/SYSKEY.TAB                 <- this was different from the on in .
   ln SYSKEY.TAB ../xcd

   Now, both directories are using the same set of SCHEMA, ROUTE.SYS, and
   SYSKEY.TAB

2) I noticed that the majority of the XCD-produced MD files (surface, upper
   air, synoptic, etc.) were bad.  Their file sizes were:

ls -alt MDXX*
-rw-rw-r--    1 ldm      udata     4989748 Jul 24 16:16 MDXX0035
-rw-rw-r--    1 ldm      udata     4149660 Jul 24 16:16 MDXX0065
-rw-rw-r--    1 ldm      udata     5078740 Jul 24 15:43 MDXX0034
-rw-rw-r--    1 ldm      udata        6656 Jul 24 02:38 MDXX0045
-rw-rw-r--    1 ldm      udata     5979224 Jul 24 01:08 MDXX0064
-rw-rw-r--    1 ldm      udata     4987048 Jul 23 23:59 MDXX0033
-rw-rw-r--    1 ldm      udata        6656 Jul 23 23:47 MDXX0005
-rw-rw-r--    1 ldm      udata        6656 Jul 23 23:17 MDXX0015
-rw-rw-r--    1 ldm      udata        6656 Jul 23 23:17 MDXX0025
-rw-rw-r--    1 ldm      udata        6656 Jul 23 23:14 MDXX0055
-rw-rw-r--    1 ldm      udata        6656 Jul 23 14:46 MDXX0044
-rw-rw-r--    1 ldm      udata        6656 Jul 23 12:23 MDXX0014
-rw-rw-r--    1 ldm      udata        6656 Jul 23 12:23 MDXX0024
-rw-rw-r--    1 ldm      udata        6656 Jul 23 12:20 MDXX0004
-rw-rw-r--    1 ldm      udata        6656 Jul 23 12:20 MDXX0054
-rw-rw-r--    1 ldm      udata     4671164 Jul 22 18:14 MDXX0063

   You can see from this partial, edited listing that a number of the
   files have a size of 6656.  This indicates that they were created
   incorrectly.  I deleted all of the XCD-created MD files (while the
   LDM was turned off), so they could get recreated correctly.


Continuing on:

<still as 'mcidas'>

batch.k XCD.BAT          <- this and the next step came after 1) and 2)
batch.k XCDDEC.BAT          below
After making the above changes, I restarted the LDM:

<as 'ldm'>
ldmadmin start

Now, as new MD files are coming in and being decoded, they look
properly sized:

ls -alt MDXX* 
-rw-rw-r--    1 ldm      udata    32688636 Jul 24 16:34 MDXX0005
-rw-rw-r--    1 ldm      udata     4552888 Jul 24 16:34 MDXX0035
-rw-rw-r--    1 ldm      udata     6460512 Jul 24 16:34 MDXX0055
-rw-rw-r--    1 ldm      udata     4155336 Jul 24 16:34 MDXX0065
-rw-rw-r--    1 ldm      udata     3526680 Jul 24 16:29 MDXX0015
-rw-rw-r--    1 ldm      udata       16640 Jul 24 16:22 MDXX0025

The munged MD files was _probably_ the sticking point for the creation
of the *.RAT files.  If the decoder can't correctly write to the output
MD file, the rapid access files don't get updated.  In fact, as I write
this I note that *.RAT files are now being produced/updated.  Yahoo!

Another problem I just saw is that the McIDAS-XCD output directory is
apparently being scoured by LDM's scour utility.  This is not advised
since scour can delete files that are open for writing by XCD routines.
I strongly recommend that the LDM scour of this directory be stopped.
In fact, I stopped it by editing ~ldm/etc/scour.conf and commenting
out ALL lines that scoured things in subdirectories of /var/data/ldm/mcidas.
Since there is already a cron entry for running the McIDAS scour routine,
mcscour.sh, your system should be in no danger of filling the disk.

While I was on foxfire, I took the liberty of doing a couple of
other things:

1) upgraded McIDAS-X to addendum 7.806; this was not necessary to get
   the decoding working, but, what the hey!

2) updated ~ldm/decoders/xcd_run with a needed patch (after making changes
   to xcd_run, I stopped and restarted the LDM)

3) deleted all FSUS* files from /var/data/ldm/mcidas/xdata (these were not
   getting scoured)

4) commented out the ~ldm/etc/pqact.conf entries for filing front analysis
   and forecast files (ASUS* and FSUS*).  McIDAS no longer uses these
   files; it reads the frontal information from the XCD-produced text
   spool files (/var/data/ldm/mcidas/xcd/*.XCD)

As I close, I think that everything is running correctly.  Please let
me know if you see anything amiss.

Tom

>From address@hidden Wed Jul 24 11:53:24 2002

Tom,

I sure feel much better now. Thanks so much.

This was getting quite frustrating, since I invested a lot of time and
didn't see anything wrong. Live and learn, I guess.

Thanks again, Bill

< Bill Fingerhut, Professor           PHONE: 802-626-6257 >
< Meteorology Dept                    FAX:   802-626-9770 >
< Lyndon State College                                    >
< Lyndonville, Vt 05851                                   >
<                                                         >
< EMAIL:     address@hidden                >
<            address@hidden                  >
< WWW:       http://apollo.lsc.vsc.edu/                   >
<                                                         >
< disclaimer: I know nothing - I only work here.          >