20000707: ROUTE PostProcess BATCH invocations and general setup (cont.)

>From: "James D. Marco" <address@hidden>
>Organization: Cornell
>Keywords: 200007062021.e66KLJT26042 McIDAS-X ROUTE PP BATCH

Jdm,

>OK.  I created a new file: NEWROUTE.BAT and rebuilt the routing.

OK, I see that the various image products now have ROUTE PostProcesses
setup.

>However, there was still no post process stuff defined for the TOPOs
>I added in a couple and rebuilt again.

The compositing of the topography and Unidata-Wisconsin images is done
through the PostProcess BATCH file for the image products coming in
the UW datastream (e.g., GE-VIS.BAT specified for the GOES-East VIS
product, UV, etc.).  Your adding execution of PostProcesses for the
created composites created an infinite processing loop.  Here is an
example of what happened:

      a GOES-East Vis product arrives
      it is decoded by lwtoa3
      the PostProcess BATCH file GE-VIS.BAT is run by lwtoa3
 ....>GE-VIS.BAT composites topography and the GOES Vis image
 .               composites the the vis image with the GOES-West vis image
 .               (the West image comes in before the East image)
 .....you setup the N2 product, GOES-East VIS/TOPO Composite, to also run
      GE-VIS.BAT.  When the composite got finished, GE-VIS.BAT was run
      again

By the time I noticed this, there were about 100 invocations of batch.k
running.  I had to SUSpend the N2 product (GOES-East VIS/TOPO Composite)
and wait until all of the processing concluded before proceeding.

I then edited NEWROUTE.BAT and removed the PP= specifications on all
of the topographic composite products (N1-N8), and then remade the
routing table (batch.k NEWROUTE.BAT).

All-in-all, it was pretty humorous seeing the number of batch.k invocations
rise.  Your machine is pretty fast, so there were lots of them :-)

>The PP batch files now appear in 
>the ROUTE list.  I am not sure which post process files belong to which.
>The East/West composite is an example.  Do you have a list on the web 
>somewhere?

The copy of DROUTE.BAT I send in the distribution has the recommended
setup.

>       Any how.  THANK YOU for your efforts and time.

No problem.

>I need to go,
>so we'll pick this up later... (Dinner engagement with you-know-who
>as the chief cook on the grill.)

I understand completely.

>       I'll set up the mail alarms when I get home.

As I am writing this, I see that Unidata-Wisconsin products are coming
in, and PostProcesses are creating the composites.  Here are snippits
from the routing table listing:

route.k LIST
S Pd         Description         Range       Last      Received  Post Process C
- -- ------------------------- --------- ------------ ---------- ------------ -
  CA Cloud Top Pressure        1100-1109 AREA1100     00189 2116     none     3
  CB Precipitable Water        1110-1119 AREA1110     00189 2131     none     3
  CC Sea Surface Temperature   1120-1129 AREA1120     00189 2133     none     3
  CD Lifted Index              1130-1139 AREA1131     00189 2134     none     3
  CE CAPE                      1140-1149 AREA1141     00189 2135     none     3
  CF Ozone                     1150-1159 AREA1151     00189 2151     none     3
  CI GOES-E/W IR Composite       80-89   AREA0080     00189 2137     none     3
  CV GOES-E/W VIS Composite      90-99   AREA0091     00189 2139     none     3
  CW GOES-E/W H2O Composite      70-79   AREA0070     00189 2137     none     3
  LD NLDN Lightning Flashes      71-71       none        none        none     3
  MA Surface MD data            default      none        none    SFC.BAT      3
  N1 GOES-East IR/TOPO Composi  220-229  AREA0220     00189 2137     none     3
  N2 GOES-East VIS/TOPO Compos  230-239  AREA0230     00189 2139     none     3
  N3 GOES-West IR/TOPO Composi  240-249  AREA0240     00189 2117     none     3
  N4 GOES-West VIS/TOPO Compos  250-259  AREA0250     00189 2117     none     3
  N5 MDR/TOPO Composite         260-269  AREA0260     00189 2106     none     3
  N6 Mollweide IR/TOPO Composi  270-279      none        none        none     3

  ...

  U1 Antarctic IR Composite     190-199  AREA0190     00189 2110     none     3
  U2 FSL2 hourly wind profiler  default  MDXX0089     00189 2117     none     3
  U3 Manually Digitized Radar   200-209  AREA0200     00189 2106 MDR.BAT      3
  U5 GOES-West US IR Band 4     130-139  AREA0130     00189 2117 GW-IR.BAT    3
  U6 FSL2 6-minute Wind profil  default  MDXX0099     00189 2204     none     3
  U9 GOES-West US Visible       120-129  AREA0120     00189 2117 GW-VIS.BAT   3
  UA Educational Floater I      160-169  AREA0160     00189 2131     none     3
  UB GOES-West US Water Vapor   170-179  AREA0170     00189 2116 GW-WV.BAT    3
  UC Educational Floater II      60-69   AREA0060     00189 2134     none     3
  UI GOES-East US IR Band 4     150-159  AREA0150     00189 2137 GE-IR.BAT    3
  UR Research Floater           180-189      none        none        none     3
  US Undecoded SAO Data         default      none        none        none     1
  UV GOES-East US Visible       140-149  AREA0140     00189 2139 GE-VIS.BAT   3
  UW GOES-East US Water Vapor   210-219  AREA0210     00189 2137 GE-WV.BAT    3
  UX Mollweide Composite IR     100-109      none        none    MOLL.BAT     3
  UY Mollweide Composite H2O    110-119      none        none        none     3
route.k: Done


Now, on to the other things I wanted to touch base on.

First off, you system is working, so none of the changes I am going to
suggest are super critical.  I believe, however, that if you make the
changes, you life will be easier in the future.

1) My recommendation is for each McIDAS-X user to have his/her own
   McIDAS working directory.  For the user 'mcidas' (the super user
   for McIDAS-X, -XCD), this would be ~mcidas/workdata; for all other
   users, this would be ~<user>/mcidas/data.

   The reason that each user should have his/her own McIDAS working
   directory is so that they can create/manipulate a set of files
   without affecting other users.  The most important of these files
   is the user's file of McIDAS REDIRECTions, LWPATH.NAM.

   When things are configured as per my recommendations, the following
   command will result in the finding of only one copy of LWPATH.NAM:

   DMAP LWPATH.NAM FORM=ALL

   This command run from your 'mcidas' account now results in:

dmap.k LWPATH.NAM FORM=ALL
PERM      SIZE LAST CHANGED FROM          PATHNAME
---- --------- ------------ ------------  --------
-rw-      4390 Jul 06 15:55 R LWPATH.NAM  ./LWPATH.NAM
-rw-      4390 Jul 06 15:55 M            #/home/mcidas/data/LWPATH.NAM
-rw-      4390 Jul 06 15:55 M            #/home/mcidas/data/LWPATH.NAM
-rw-      4390 Jul 06 15:55 M            #/home/mcidas/data/LWPATH.NAM
-rw-      4390 Jul 06 15:55 M            #/home/mcidas/data/LWPATH.NAM
-rw-      4369 Apr 19 08:46 M            #/unidata/xcd/LWPATH.NAM
-rw-      4369 Apr 19 08:46 M            #/var/data/mcidas/LWPATH.NAM
-rw-      4369 Apr 19 08:46 M            #/var/data/xcd/LWPATH.NAM
35057 bytes in 8 files

   OK, so in this listing there is multiple listings of the same file
   (/unidata/xcd/LWPATH.NAM is the same file as /var/data/mcidas/LWPATH.NAM
   and /var/data/xcd/LWPATH.NAM), but the point is that there is more
   than one LWPATH.NAM that can be found through MCPATH conventions (this
   is what the 'M' in the FROM column means).  The reason that there is
   more than one listing of the same file, /home/mcidas/data/LWPATH.NAM,
   is that the /home/mcidas/data directory is specified more than once
   in the user 'mcidas' MCPATH.

   The copy of LWPATH that is used is the one first in the list, and
   that one is found by virtue of a REDIRECTion (the 'R' in the FROM
   column).  If the REDIRECTion ever got deleted, then the copy in
   /home/mcidas/data would get used.  If that copy got deleted (there
   should never be a copy of LWPATH.NAM in ~mcidas/data), then the
   copy in /unidata/xcd would be used, and so on.

   Again, each user should have access to one and only one copy of
   LWPATH.NAM, and that copy should be in his/her own McIDAS working
   directory.

2) Each user should specify their McIDAS working directory in the definition
   of their MCDATA environment variable:

   setenv MCDATA $HOME/workdata     -  for the user 'mcidas'  (C shell syntax)
   setenv MCDATA $HOME/mcidas/data  -  for all other users    (C shell syntax)

3) The MCPATH for each user should contain their McIDAS working directory
   as the first directory:

   setenv MCHOME home_directory_of_the_user_mcidas
   setenv MCPATH ${MCDATA}:$MCHOME/data:$MCHOME/help:...

4) The configuration files for McIDAS-XCD get installed into the workdata
   subdirectory of the user 'mcidas'.  This is where they should stay.
   In particular, they should not be copied to any directory in the
   MCPATH of the user 'mcidas'.  Right now, you have XCD configuration
   files in both ~mcidas/workdata and in /var/data/mcidas (which is a
   link to /var/data/xcd).

   The reason you don't want these configuration files in any directory
   other than ~mcidas/workdata is that you do not want any other user
   mucking with the files.  Given that the ~mcidas/data and ~mcidas/help
   directories should be included in each user's MCPATH, it would be
   easy for these users to, intentionally or not, modify one or more
   of the configuration files in ~mcidas/data, ~mcidas/help, or
   in your case /var/data/mcidas.

5) There should be no McIDAS environment variables defined in the 'ldm'
   account.  Currently, you have blank definitions for McINST_ROOT, MCDATA,
   MCPATH, etc. in your 'ldm' account.  I strongly recommend removing
   these from your shell definition file (which appears to be .bashrc).
   These environment variables get defined in scripts run by the user
   'ldm' when needed (batch.k, mcscour.sh, and xcd_run).

6) The scripts batch.k, mcscour.sh, and xcd_run which are located in the
   ~ldm/decoders directory on your system should be edited to change the
   definition of MCDATA to ~mcidas/workdata (instead of ~mcidas/data).

7) I would move all of the BATCH files that you have created in ~mcidas/data
   to the ~mcidas/workdata directory IF they are designed to only be
   used by the user 'mcidas'.  This way, they will not be in any
   user's MCPATH (since no user's MCPATH should contain the MCDATA directory
   of any other user).

8) After making the above changes, I would do some cleanup in the
   ~mcidas/data and /var/data/mcidas directories.  In particular, I would
   remove files in those directories that are also located in the
   ~mcidas/workdata directory.  We can get into specifics about which
   these files are if/when you decide to make the changes.

There may be a couple of other small things to address, but I think
that they can wait until after these other changes are made.

The thing that I am most concerned with is the amount of extra work
that you may have to do in order to install new McIDAS versions.  The
setup I recommend minimizes such work, so I highly recommend it.

Talk to you later...

Tom

>From address@hidden Fri Jul  7 21:40:31 2000
>Subject: Re: 20000707: ROUTE PostProcess BATCH invocations and general setup 
>(cont.)

Tom,
        Ooops.  'thought I was getting a handle on this.  Sorry for the
mung. Thanks for the cleanup.


At 04:15 PM 7/7/2000 -0600, you wrote:
<deleted...>
>The compositing of the topography and Unidata-Wisconsin images is done
>through the PostProcess BATCH file for the image products coming in
>the UW datastream (e.g., GE-VIS.BAT specified for the GOES-East VIS
>product, UV, etc.).  Your adding execution of PostProcesses for the
>created composites created an infinite processing loop.  Here is an
>example of what happened:
>
>      a GOES-East Vis product arrives
>      it is decoded by lwtoa3
>      the PostProcess BATCH file GE-VIS.BAT is run by lwtoa3
> ....>GE-VIS.BAT composites topography and the GOES Vis image
> .               composites the the vis image with the GOES-West vis image
> .               (the West image comes in before the East image)
> .....you setup the N2 product, GOES-East VIS/TOPO Composite, to also run
>      GE-VIS.BAT.  When the composite got finished, GE-VIS.BAT was run
>      again

Ouch.

>By the time I noticed this, there were about 100 invocations of batch.k
>running.  I had to SUSpend the N2 product (GOES-East VIS/TOPO Composite)
>and wait until all of the processing concluded before proceeding.
>
>I then edited NEWROUTE.BAT and removed the PP= specifications on all
>of the topographic composite products (N1-N8), and then remade the
>routing table (batch.k NEWROUTE.BAT).
>
>All-in-all, it was pretty humorous seeing the number of batch.k invocations
>rise.  Your machine is pretty fast, so there were lots of them :-)
I'm glad you have a sense of humor.  It IS funny though. The faster
the machine, the better it can run my mistakes.    

>>The PP batch files now appear in 
>>the ROUTE list.  I am not sure which post process files belong to which.
>>The East/West composite is an example.  Do you have a list on the web 
>>somewhere?
>
>The copy of DROUTE.BAT I send in the distribution has the recommended
>setup.
OK. I saw that. I need to add some auto-update stuff to the batch files
for the 'static' displays in the Map room.

>>      Any how.  THANK YOU for your efforts and time.
>As I am writing this, I see that Unidata-Wisconsin products are coming
>in, and PostProcesses are creating the composites.  
<deleted...>
GREAT!!!!

>Now, on to the other things I wanted to touch base on.
>
>First off, you system is working, so none of the changes I am going to
>suggest are super critical.  I believe, however, that if you make the
>changes, you life will be easier in the future.
I'm sure. I think I finally have the time (till the end of the month) to
go through some of the things correctly.  

>1) My recommendation is for each McIDAS-X user to have his/her own
>   McIDAS working directory.  For the user 'mcidas' (the super user
>   for McIDAS-X, -XCD), this would be ~mcidas/workdata; for all other
>   users, this would be ~<user>/mcidas/data.
Yup.  I started setting this up in /etc/skel so that each new user would 
automatically get the correct setup. I installed default $HOME/mcidas/data
in the system startup (/etc/bashrc) but got the new machines in, before the
configuration was stabilized. This results in double paths (mcidas account
for example) for pre-existing accounts. You saw the results below. It works,
but not very well. It needs to be streamlined just for easier administration.
 
>   The reason that each user should have his/her own McIDAS working
>   directory is so that they can create/manipulate a set of files
>   without affecting other users.  The most important of these files
>   is the user's file of McIDAS REDIRECTions, LWPATH.NAM.
Yes, This has happened a couple of times in the past. This will be solved
by the above configuration, I hope.  

>   When things are configured as per my recommendations, the following
>   command will result in the finding of only one copy of LWPATH.NAM:
Yes.  Much easier to administer and more flexable. 

>   DMAP LWPATH.NAM FORM=ALL
>   This command run from your 'mcidas' account now results in:
<deleted...>
<...deleted>
>   OK, so in this listing there is multiple listings of the same file
>   (/unidata/xcd/LWPATH.NAM is the same file as /var/data/mcidas/LWPATH.NAM
>   and /var/data/xcd/LWPATH.NAM), but the point is that there is more
>   than one LWPATH.NAM that can be found through MCPATH conventions (this
>   is what the 'M' in the FROM column means).  The reason that there is
>   more than one listing of the same file, /home/mcidas/data/LWPATH.NAM,
>   is that the /home/mcidas/data directory is specified more than once
>   in the user 'mcidas' MCPATH.
Yup. Once in the system /etc/bashrc, once in the users local .bashrc.
>   The copy of LWPATH that is used is the one first in the list, and
>   that one is found by virtue of a REDIRECTion (the 'R' in the FROM
>   column).  If the REDIRECTion ever got deleted, then the copy in
>   /home/mcidas/data would get used.  If that copy got deleted (there
>   should never be a copy of LWPATH.NAM in ~mcidas/data), then the
>   copy in /unidata/xcd would be used, and so on.

>   Again, each user should have access to one and only one copy of
>   LWPATH.NAM, and that copy should be in his/her own McIDAS working
>   directory.
This has been a problem in the past, since it meant 14 iterations of 
configuring McIDAS on each of the potential McIDAS servers, for each 
user account on a machine.  I resisted centralized accounts till I got the
internal network infrastructure built (was AppleTalk, with some 10BT, now
100BTX with some 10BT).  This was completed in the middle of the Fall 
semester. Funding was made available for 2 dual processor servers and 3 
single processor McIDAS machines, and user account upgrades were delayed 
till I had the new servers, that is until now. Your advice is invaluable 
since I have the time to do it right as I configure the new systems, now
that they are set up. I'll retrofit the old machines when I add the student
accounts in August.
    
>2) Each user should specify their McIDAS working directory in the definition
>   of their MCDATA environment variable:
>   setenv MCDATA $HOME/workdata     -  for the user 'mcidas'  (C shell
syntax)
>   setenv MCDATA $HOME/mcidas/data  -  for all other users    (C shell
syntax)
Yup.

>3) The MCPATH for each user should contain their McIDAS working directory
>   as the first directory:
>
>   setenv MCHOME home_directory_of_the_user_mcidas
>   setenv MCPATH ${MCDATA}:$MCHOME/data:$MCHOME/help:...

>4) The configuration files for McIDAS-XCD get installed into the workdata
>   subdirectory of the user 'mcidas'.  This is where they should stay.
>   In particular, they should not be copied to any directory in the
>   MCPATH of the user 'mcidas'.  Right now, you have XCD configuration
>   files in both ~mcidas/workdata and in /var/data/mcidas (which is a
>   link to /var/data/xcd).
This was a hack to solve other pathing problems. 
>   The reason you don't want these configuration files in any directory
>   other than ~mcidas/workdata is that you do not want any other user
>   mucking with the files.  Given that the ~mcidas/data and ~mcidas/help
>   directories should be included in each user's MCPATH, it would be
>   easy for these users to, intentionally or not, modify one or more
>   of the configuration files in ~mcidas/data, ~mcidas/help, or
>   in your case /var/data/mcidas.
Yes, not good.

>5) There should be no McIDAS environment variables defined in the 'ldm'
>   account.  Currently, you have blank definitions for McINST_ROOT, MCDATA,
>   MCPATH, etc. in your 'ldm' account.  I strongly recommend removing
>   these from your shell definition file (which appears to be .bashrc).
>   These environment variables get defined in scripts run by the user
>   'ldm' when needed (batch.k, mcscour.sh, and xcd_run).
This was done to CLEAR the general system definitions. As I say, this
solved the multiple individual configuration problems, but complicated 
McIDAS and LDM configuration. I will need to perform an individual
configuration for the two special accounts. Hmmm, the old csh shouldn't 
read the system bashrc.  I rarely do anything in LDM or McIDAS (except 
some small batch files), sooo, I'll try switching these two accounts 
over to csh. Everyone else uses bash. The best of both worlds?

>6) The scripts batch.k, mcscour.sh, and xcd_run which are located in the
>   ~ldm/decoders directory on your system should be edited to change the
>   definition of MCDATA to ~mcidas/workdata (instead of ~mcidas/data).
Yeah, I saw that.  

>7) I would move all of the BATCH files that you have created in ~mcidas/data
>   to the ~mcidas/workdata directory IF they are designed to only be
>   used by the user 'mcidas'.  This way, they will not be in any
>   user's MCPATH (since no user's MCPATH should contain the MCDATA directory
>   of any other user).
Yup. This is on the agenda.  I will probably place user batch files in 
another directory (util?) and include it in the path. I originally did a 
locate for BAT files and just started building them in the same directory,
knowing I wasn't mucking around with the McIDAS data paths.    

>8) After making the above changes, I would do some cleanup in the
>   ~mcidas/data and /var/data/mcidas directories.  In particular, I would
>   remove files in those directories that are also located in the
>   ~mcidas/workdata directory.  We can get into specifics about which
>   these files are if/when you decide to make the changes.
Yup, I had copied the whole ~mcidas/data & ~mcidas/workdata directories
into the /var/data/mcidas directory to solve other pathing problems, 
while under the gun to get the program running in time for some classes. 
I know its a mess...and I need the wasted disk space.
   
>There may be a couple of other small things to address, but I think
>that they can wait until after these other changes are made.
>
>The thing that I am most concerned with is the amount of extra work
>that you may have to do in order to install new McIDAS versions.  The
>setup I recommend minimizes such work, so I highly recommend it.
I agree. And, I well appreciate your recommendations.  Most of this was
already planned, but it is reassuring to get it from the expert! And,
I have gained a much deeper understanding of what goes on in McIDAS as
a fringe.

>Talk to you later...
Absolutly. Once McIDAS is up to spec, there is this thing out there
called McADDE, that looks to be a handy piece of machina ...
How many brews is this gonna' cost me, anyhooo?....Well Worth It!! 
                                        jdm 
>
>Tom
>***************************************************************************
* <
>Unidata User Support                                    UCAR Unidata
Program <
>(303)497-8644                                                  P.O. Box
3000 <
>address@hidden                                   Boulder, CO
80307 <
>---------------------------------------------------------------------------
- <
>Unidata WWW Service                        http://www.unidata.ucar.edu/
  <
>***************************************************************************
* <
>
James D. Marco, address@hidden, address@hidden
Programmer/Analyst, System/Network Administration, 
Computer Support, Et Al. 
Office:                 1020 Bradfield Hall, Cornell University 
Home:                   302 Mary Lane, Varna    (607)273-9132
Computer Lab:   1125 Bradfield          (607)255-5589