20030609: McIdas MD file problems.

>From: Mark Tucker <address@hidden>
>Organization: Lyndon State College
>Keywords: 200306091657.h59GvRLd029618 McIDAS-XCD FreeBSD shared memory

Mark,

>I seem to be missing several groups of MD files in our new McIdas 2002
>installation on our ldm server Omega.  (FreeBSD 4.8, ldm 6.0.12).  The xcd
>decoders are generating MD files 0-10 and 70-100).

MD files 1-10 contain surface data decoded by XCD decoders.  MD files
70-100 contain NLDN lightning, hourly summary profiler, and 6-minute
profiler data decoded by ldm-mcidas decoders.

>The data is coming in as
>our old server Zeus is feeding from Omega and is processing the files
>correctly.  I checked the~mcidas/workdata/XCD_START.LOG and found numerous
>errors. The file has grown to over 300MB in size in the past month or so.
>Here are some of the more common error lines I'm seeing.  There are thousands
>of instances of the DMSYN and DMSFC error messages:
>
>DMSYN: mcinfoid - unable to get station
>DMMISC: m0f14dec - invalid value for flags(4) = 40
>DMMISC: m0pirdec - invalid value for flags(4) = 60
>DMSFC: skyini - invalid key name 10
>
>Any ideas about what I need to correct here?

My first guess would be that omega is not setup quite correctly for XCD
decoding.  The setup process includes (in this order)

1) make sure that the LDM is not running
2) setup McIDAS FILE REDIRECTions for the files that will be generated
   and used by XCD decoders
3) copy ~mcidas/data/SCHEMA, ~mcidas/data/SYSKEY.TAB, and
   ~mcidas/workdata/ROUTE.SYS to the directory in which you want your XCD
   decoders to create their output.  These files will have to have
   REDIRECTions setup in the McIDAS account in order for them to be
   found in this directory.  Make sure that the read/write permissions
   on these three files is such that both 'ldm' and 'mcidas' can read
   AND write them.
4) define the McIDAS string XCDDATA to be the directory you just copied
   SCHEMA, SYSKEY.TAB, and ROUTE.SYS to
5) since you have already been running for awhile, do some cleanup before
   proceeding:

   - remove the *.RAP and *.RAT files from the directory in which
     files will be decoded

   - remove the IDXALIAS.DAT file from the directory in which files
     will be decoded

   - remove DECOSTAT.DAT, SIGCO.DAT, COUNTRY.DAT, DECINFO.DAT, CIRCUIT.DAT,
     and GROUPS.DAT files from the ~mcidas/workdata directory

6) as 'mcidas' run the XCD setup BATCH files from the ~mcidas/workdata
   directory:

   cd ~mcidas/workdata
   batch.k XCD.BAT
   batch.k XCDDEC.BAT

7) now, even if you previously had McIDAS GRID decoding turned on, you will
   have to turn it on again (since DECINFO.DAT was deleted in step 6)
   assuming, of course, that you want to decode model output into
   McIDAS GRID files:

   decinfo.k SET DMGRID ACTIVE

8) at this point, you should be able to turn the LDM back on and start
   decoding data.

>I couldn't seem to find anything
>related in the email archive or documentation.

Right, your case is a bit unusual.  That is why I suspect a setup problem.

>Also, I've occasionally run into a problem with the shared memory limits on
>the default FreeBSD kernel.  I've found in the archive searches and google
>that compiling a new kernel after tweaking the following kernel parameters
>sould enable more resources for applications.  I tried increasing SHMMAXPGS to
>some high values but end up with less shared memory available after booting
>with the new kernel.

>Are there any "known good" values for mcidas & ldm setups?

I don't think that the LDM will need more than the default available
on FreeBSD.  McIDAS, on the other hand, could need substantially more
depending on the size and number of the sessions that are run.

>options         SHMMAXPGS=1025  
>options         SHMALL=1025     
>options         SHMMAX="(SHMMAXPGS*PAGE_SIZE+1)" 
>options         SHMMIN=2       
>options         SHMMNI=33  
>options         SHMSEG=9 

I found the following in a Google Groups discussion of Xfree86 shared
memory use under FreeBSD:

  > This is a plea for help.
  >
  > I used to have a problem with Gnome - it ate up all of my SysV shared
  > memory.  My window manager (sawmill) 's title bars would come up blank
  > because of this.  I eventualy found the "use MIT-SHM" checkbox in the
  > imlib settings and turned it off, and the problem mostly went away.
  >
  > I have recently upgraded to Gnome 1.2 and it has come back with a
  > vengence.  The checkbox no longer has any effect.  I have bumped up
  > the amount of shared memory, but it all gets used, no matter how much
  > is available.  It is driving me crazy.  I can't run other programs
  > (samba, fxtv) because there is never any shared memory left.  Something
  > is eating it all - gnome, gtk, imlib, I don't know how these pieces
  > fit together or exactly where the fault lies.  I am desperately looking
  > for a solution that doesn't involve just giving up this very pleasant
  > and otherwise useful software.
  >
  > I have asked a co-worker who also runs Gnome on FreeBSD to check his
  > shared memory usage, and it was fine.  The only difference is that I
  > am running -current and he has 4.0-release.
  
  Hmm, where my crystal ball... Aha, I see - probably you are using Xfree
  4.0, while your friend Xfree3.5*. It is where the problem lie (see
  below).
  
  > I can't find any evidence that I am not the only person on the planet
  > having this problem, but I am completely out of ideas.  Does anyone
  > know what's going on here?  Look at this output of "icps -mbop", it's
  > ridiculous:
  
  Some time ago I've answered question like this, so let me quote myself:
  
  Subject: Re: Shared memory changes in current?
  Date: Tue, 06 Jun 2000 15:32:19 +0300
  From: Maxim Sobolev <address@hidden>
  To: Alexander Sanda <address@hidden>
  
  "It has noting to do with kernel/gnome. XFree 4.0 is known to be very hungry 
for
  the shared memory, so you should increase SHMSEG parameter in your kernel 
config
  file. There are no guidelines as to what exact number will be sufficient, so 
you
  should define it in experimental way. I personally set it to 100 (options
  SHMSEG=100) and do not see any warnings anymore."

Also, ours system administrator will be looking into the FreeBSD shared
memory issue in the coming days.

>BTW, when will we see FreeBSD install notes in the documentation?

In v2003 which I am working on right now.

>Thanks.

Cheers,

Tom

>From address@hidden Tue Jun 10 12:30:19 2003

On Mon, 09 Jun 2003 12:14:56 -0600
Unidata Support <address@hidden> wrote:

> My first guess would be that omega is not setup quite correctly for XCD
> decoding.  The setup process includes (in this order)
 [snip]

Tom,
I'm not sure what I missed during the original setup our our ldm/mcidas on
Omega but this worked.  I recall checking each of these items during the
mcidas xcd setup but I must have skipped something in there.   This also fixed
a problem I was having with certain area files not being proplerly filed.  

Thanks again.

-- 
Mark Tucker
Meteorology Dept. Systems Administrator
Lyndon State College
http://apollo.lsc.vsc.edu
address@hidden
(802)-626-6328