20000104: McIDAS-XCD data ingest

>From: "Thomas L. Mote" <address@hidden>
>Organization: University of Georgia
>Keywords: 200001042125.OAA01383 McIDAS-XCD setup

Tom,

>I have installed McIDAS 7.6 after not having used McIDAS 
>for the past two years. (I was away from the university 
>last year.) The last version of McIDAS that I had installed 
>was 7.1.
>
>Anyway, I installed and built 7.6 in a mcidas user account. 
>Created a mcadde account and installed the ADDE server. I 
>installed the XCD decoders. Here's the problem...
>
>Right now, I am getting AREA files in my data directory 
>/data/mcidasd.

OK, the AREA files are created by the ldm-mcidas decoder, lwtoa3.
GRID, POINT, and TEXT data files are created by XCD processes.

>I am getting all sorts of .IDX files and a 
>few other files in /unidata/home/mcidas/workdata.

These files are created by XCD processes.  The reason that they are
appearing in the ~mcidas/workdata directory is that you have not
setup REDIRECTions for them.

Configuring McIDAS-XCD is presented in:

Unidata McIDAS-X HomePage
http://www.unidata.ucar.edu/packages/mcidas/mcx
  Unidata McIDAS-X Installation Instructions
  http://www.unidata.ucar.edu/packages/mcidas/mcx/mcidas-x.html
    Building, Installing, and Configuring McIDAS-XCD
    http://www.unidata.ucar.edu/packages/mcidas/mcx/mcidas-xcd.html
      Chapter 2 - McIDAS-XCD Software Installation
      http://www.unidata.ucar.edu/packages/mcidas/mcx/mcidas-xcd.html
        Installation procedures
        http://www.unidata.ucar.edu/packages/mcidas/mcx/xcd_install_procs.html
          Configuring the Unidata McIDAS-XCD files
          http://www.unidata.ucar.edu/packages/mcidas/mcx/xcd_config.html

The configuration steps presented there should be done:

o after installing McIDAS and configuring the 'mcidas' account.
  In particular, they need to be done after setting up the set of
  REDIRECTions that McIDAS-XCD routines will use to locate/read/write
  data files.  This is presented in:

  Configuring Unidata McIDAS-X Accounts
  http://www.unidata.ucar.edu/packages/mcidas/mcx/config_mcx.html
    Configuring the mcidas Account
    http://www.unidata.ucar.edu/packages/mcidas/mcx/config_mcidas.html
    
  A short recap of the setting up of REDIRECTions necessary for the
  'mcidas' account is given below as well.

o while the LDM is not running

The Unidata McIDAS-X distributions are sent out with an example
REDIRECTion file, EXAMPLE.NAM, that:

o exists in the ~mcidas/data directory

o is designed to be copied to ~mcidas/data/LOCAL.NAM

o LOCAL.NAM should be edited by the person installing McIDAS; the
  changes that are needed are changing the directory for data files
  to wherever you want to store them on your system.  From what you
  note above, this looks to be the /data/mcidasd directory on your
  system.

o the REDIRECTions in LOCAL.NAM need to be made active in the 'mcidas'
  account by:

  o starting a McIDAS-X session as the user 'mcidas'
  o making sure that the file LWPATH.NAM either does not yet exist
    or is located in the ~mcidas/workdata directory.  If it does not
    already exist, keep going.  If it does exist and is in the
    ~mcidas/workdata directory, keep going.  If it already exists
    and is NOT in the ~mcidas/workdata directory, EXIT McIDAS and
    delete it from wherever it was found.  You can find out if
    it exists using the McIDAS DMAP command:

    DMAP LWPATH.NAM

  o if you had to EXIT McIDAS and delete LWPATH.NAM in the step
    above, restart McIDAS.

  o restore the REDIRECTions in LOCAL.NAM from the McIDAS command window:

    REDIRECT REST LOCAL.NAM

After restoring the REDIRECTions from LOCAL.NAM, XCD files can be
written into the correct output directory after you stop and restart
the LDM.  The reason for this is that the REDIRECTions that were in
place when the LDM was run will not be changed until the XCD processes
are stopped and restarted.  The easiest way to accomplish this is to
stop and start the LDM.  Before restarting the LDM, please read on.

I will venture to guess that since your installation processes missed
the creating and editing of LOCAL.NAM and restoring of its
REDIRECTions, you probably did not run the XCD setup McIDAS BATCH
scripts, XCD.BAT and XCDDEC.BAT.  If this is the case, there are other
steps that you should follow _BEFORE_ restarting your LDM.  Basically,
the step that must be taken is the running of 'BATCH XCD.BAT' and
'BATCH XCCDEC.BAT' from the McIDAS Command window in a McIDAS session
run as the user 'mcidas' while the LDM is not running and after
REDIRECTions have been setup using LOCAL.NAM described above.

After stopping and restarting the LDM, you should verify that the
XCD-produced files (e.g. *.IDX etc.) are now being created in the
directory you specified for them in ~mcidas/data/LOCAL.NAM.

Only after the XCD process are correctly creating output data files
in your intended directory (/data/mcidasd), you can do some cleanup
in the 'mcidas' working directory, ~mcidas/workdata.  The cleanup will
consist of deleting the XCD-produced files that belong in the
output data directory, /data/mcidasd.
 
>I assume I should have all of the data in /data/mcidasd.

While this is not mandatory, it is recommended to put all McIDAS data
files in a single directory.  This makes disk management easier.

>I really don't fully understand how XCD works nor what each of 
>these data files includes.

I admit that the process is fairly lengthy.  One reason for this is
that XCD was designed at SSEC to read data from serial ports.  In
the Unidata world, users have the XCD processes read data files from
the LDM.

>I have been following the 
>instructions on the Unidata web site as closely as 
>possible, simply changing pathnames where appropriate.

Sounds good.  It seems like maybe one step was missed.  I must quickly
add that if the REDIRECTions were not setup in LOCAL.NAM and made
active, then the other XCD configuration steps like running 'BATCH
XCD.BAT' and 'BATCH XCDDEC.BAT' will not have been completed
correctly.  This is why I suggest stopping the LDM and going back and
getting the preliminary steps done correctly first and then
proceeding.

>As you would expect, I am able to view satellite imagery, 
>but I cannot get any point, grid or text data.

Right.  Again, the imagery is being handled by ldm-mcidas decoders.
They have nothing to do with XCD.  The point, grid, and text data are
all created by XCD processes.

>I really don't even know where to start. Any suggestions?

I think that you are very close to being finished.  The best thing
is to drop back and review the section on the setup of REDIRECTions
outlined above and in the online docs and then move forward redoing
the steps covered in the configuration of XCD.

By the way, as far as you having setup the ADDE remote server, it
looks like this was not done correctly or did not "take" IF the
machine on which you are doing this is cacimbo.ggy.uga.edu.  The
reason I say this is that when I try to do a:

telnet cacimbo.ggy.uga.edu 500

(ADDE uncompressed data transfers are done through port 500)

I get:

Trying...
telnet: connect: Connection refused

If the remote ADDE server installation steps were completed, then
outside machines would be able to connect to port 500, using telnet
(even though no telnet transactions would be allowed).  So, you
will probably look into the ADDE installation steps again (IF
the machine name really is cacimbo.ggy.uga.edu 500).

>Thanks.

If you run into XCD configuration problems that you can not sort out,
I am willing to take a quick look if you provide me the logins as
the users 'mcidas' and 'ldm' on your machine.

Tom Yoksas