[McIDAS #ZIF-240347]: McIDAS frmsave.k destination directory?

Hi Darrin,

re:
> Thanks for the quick response.

No worries.

> Your explanation of how frmsave uses
> MCPATH ("first _writable_ directory") led me to the answer.
> 
> It turns out I had a subtle directory permissions problem. The ldm
> user (who was running the batch post-processing) didn't have write
> perms to the ~mcidas/workdata dir.

I thought about inquiring about the directory read/write status for 'ldm',
but I thought it might sidetrack the focus on full setting of McIDAS
environment variables.

> I think this was a side effect of
> working half with a new installation and half with a legacy directory
> tree. At some point when creating and/or copying dirs, the perms got
> messed up.

Been there; done that :-)

> For the next McIDAS release, it would be nice if there was some sort
> of error message about this (e.g. frmsave could output something along
> the lines of "Unable to write to Destination Directory, writing
> to .mctmp instead"). An error message like that probably would have
> led me to a solution sooner.

I am afraid that writing to the .mctmp directory because no other
directory in one's MCPATH is writable is not an error in the
overall scheme of things in McIDAS.  The implicit concept in
McIDAS is that it will write to the first writable directory in one's
MCPATH.  What is not intuitively obvious is that one's MCPATH gets
extended by McIDAS to include the ~/.mctmp/<ipc handle> directory
as the last directory in one's MCPATH.  This is so that the system can create
some files in a hidden directory needed for a McIDAS "session" and
then delete the files and hidden directory when that "session" ends.
(A McIDAS session is created by 'mcenv', so running 'mcenv' from
a script creates a "session".) The McIDAS Users Guide Appendix G contains
information on setting MCPATH and how it is extended and used.

One of the topics I cover in detail in McIDAS training workshops is the
ins and outs of MCPATH especially in its 'gotchas'.  Unfortunately,
what you ran into is one of MCPATH's 'gotchas'.  Here is another one:

Suppose you have a file with the same name in more than one directory
contained in your MCPATH, and further suppose that the first MCPATH
directory containing one of the files is not writable by the user
running the McIDAS session (your exact situation), but the second
one is and both directories are set to be readable.  If you run
a McIDAS command to delete the particular file, the one in the
second directory will be deleted, not the one in the first directory.
Further suppose that the goal of the process is to delete a file
and replace it with a newly created one (e.g., by an IMGCOPY).  What
will happen is that the delete will work, but the replace will
not as McIDAS will be trying to work on the file in the first MCPATH
directory where it can read the file, but not write it.  This
situation is completely unexpected by the user, and so hard to
diagnose.  MCPATH is an interesting idea, but not as foolproof as
it should be!

McIDAS provides a second file location facility call REDIRECT.  A
McIDAS REDIRECTion is a directive to read/write files into a particular
directory regardless of one's MCPATH setting.  It is easiest to explain
REDIRECT with an example:

Suppose you want to always read/write/access a file named AREA0001 in
the directory /data/mcidas.  You can create a McIDAS REDIRECTion that
will cause this to happen:

REDIRECT ADD AREA0001 "data/mcidas  (redirect.k ADD AREA0001 \"/data/mcidas)

The REDIRECTion rule gets written into the file LWPATH.NAM.  The
rule will be used by subsequent McIDAS sessions as long as the same
copy of LWPATH.NAM is used.

REDIRECTion rules can be quite a bit more complex than the simple
example above.  For instance, supposed you want to read/write/access
files whose names match the regular expression AREA000* from the
/data/mcidas directory.  Here is the REDIRECT invocation that
will set this up:

REDIRECT ADD AREA000* "/data/mcidas  (redirect.k ADD AREA000\* \"/data/mcidas)

REDIRECTions are hierarchical.  Suppose that you had defined the the
following three file REDIRECTions:

REDIRECT ADD AREA0001 "/data/mcidas
REDIRECT ADD AREA000* "/data/scratch
REDIRECT ADD AREA*    "/data/otherfiles

When these REDIRECTions are in scope for the McIDAS session, the first one
will be used for the file AREA0001 because it is the most specific.
Likewise, the second will be used for the file AREA0008 because it is
more specific than the third rule.

Basically, one uses file REDIRECTions in conjunction with MCPATH to
locate files exactly where desired.

I know that the McIDAS MCPATH and REDIRECT concepts are not as easily
grasped as they should be given their importance, but they are an
integral part of McIDAS nonetheless (sigh).  This is why I spend 
a LOT of time discussing the concepts during the McIDAS training
sessions that I put on (again, BIG sigh).

By the way, the Unidata McIDAS distribution contains two example
Bourne shell scripts that are designed to be copied to a file
with a different name; edited; and used to run McIDAS routines
outside of an interactive McIDAS session.  The files 'mcrun.sh'
(run a sequence of McIDAS commands like you are doing in your
script) and 'mcbatch.sh' (run commands from a McIDAS BATCH file)
can be found in the ~mcidas/data directory after the installation
has been installed (like yours).  You will see that I tried to
document what is going on so the end user will understand what is
needed when running McIDAS "sessions" from scripts.

> FYI, I've included my version info below.

re: McIDAS Version
> 2009 Unidata 090710    X

Very good.  I will be releasing an addendum for v2009 in the
coming weeks (before Thanksgiving for sure).

re: OS

> Linux ocs1.OCE.ORST.EDU 2.6.18-128.7.1.el5 #1 SMP Mon Aug 24 08:21:56  
> EDT 2009 x86_64 x86_64 x86_64 GNU/Linux

> (This is CentOS)

Very good.  We run Fedora here in the UPC, so CentOS is a known quantity.

re: which LDM version

> hostname:              ocs1.OCE.ORST.EDU
> os:                    Linux
> release:               2.6.18-128.7.1.el5
> ldmhome:               /usr/local/ldm
> LDM version:           6.8.1
> PATH:                  /usr/local/ldm/bin:/usr/local/ldm/decoders:/usr/ 
> local/ldm/util:/usr/local/ldm/bin:/usr/kerberos/bin:/usr/local/bin:/ 
> bin:/usr/bin:/usr/bin:/usr/local/bin:/usr/gnu/bin:/usr/sbin:.
> bin path:              /usr/local/ldm/bin
> etc path:              /usr/local/ldm/etc
> log path:              /usr/local/ldm/logs
> data path:             /usr/local/ldm/data
> LDM conf file:         /usr/local/ldm/etc/ldmd.conf
> pqact(1) conf file:    /usr/local/ldm/etc/pqact.conf
> scour(1) conf file:    /usr/local/ldm/etc/scour.conf
> product queue:         /usr/local/ldm/data/ldm.pq
> queue size:            500M bytes
> queue slots:           default
> pqsurf(1) path:        /usr/local/ldm/data/pqsurf.pq
> pqsurf(1) size:        2000000
> IP address:            0.0.0.0
> port:                  388
> PID file:              /usr/local/ldm/ldmd.pid
> Lock file:             /usr/local/ldm/.ldmadmin.lck
> ldmadmin(1) conf file: /usr/local/ldm/etc/ldmadmin-pl.conf
> maximum clients:       256
> maximum latency:       3600
> time offset:           3600
> log file:              /usr/local/ldm/logs/ldmd.log
> numlogs:               7
> log_rotate:            1
> netstat:               /bin/netstat -A inet -t -n
> metrics file:          /usr/local/ldm/logs/metrics.txt
> metrics files:         /usr/local/ldm/logs/metrics.txt*
> num_metrics:           4
> check time:            1
> delete info files:     0
> ntpdate(1):            /usr/sbin/ntpdate
> ntpdate(1) timeout:    5
> time servers:          ntp.ucsd.edu ntp1.cs.wisc.edu ntppub.tamu.edu  
> otc1.psu.edu timeserver.unidata.ucar.edu
> time-offset limit:     10
> check time abort:      0

Excellent, you are running the latest LDM!

Thanks for the information!
Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: ZIF-240347
Department: Support McIDAS
Priority: Normal
Status: Closed