[IDD #WFA-619667]: Request for CONDUIT feed

Hi Martha,

re:
> I see the following messages in ldmd.log when I try to do the local
> notifyme (that is, notifyme -vl- -f CONDUIT)
> 
> Feb 17 13:25:52 noaapxcd localhost.localdomain(noti)[2133] NOTE:
> Starting Up(6.6.5/5): 20090217130932.600 TS_ENDT {{CONDUIT,  ".*"}}
> Feb 17 13:25:52 noaapxcd localhost.localdomain(noti)[2133] NOTE: topo: 
> localhost.localdomain CONDUIT
> Feb 17 13:30:56 noaapxcd localhost.localdomain(noti)[2133] ERROR: 
> nullproc5(localhost.localdomain): RPC: Unable to receive
> Feb 17 13:30:56 noaapxcd localhost.localdomain(noti)[2133] NOTE: Exiting

The 'NOTE: topo' line shows that a 'notifyme' request was received and allowed.
The 'ERROR: nullproc5' and 'NOTE: Exiting' lines would occur when you kill
your 'notifyme' command.  These are normal and expected.
 
> I looked a bit in the UNIDATA archives, and the only thing I found was
> about the "allow" line in ldmd.conf, but we left that "as is" , i.e.,
> 
> # Under no circumstances comment out the next allow entry to localhost
> # The LDM will NOT start if the lines are commented out.
> allow ANY 
> ^((localhost|loopback)|(127\.0\.0\.1\.?$)|([a-z].*\.unidata\.ucar\.edu\.?$))

Yes, this line in ldmd.conf is what allows the localhost and machines in
the unidata.ucar.edu to connect to your LDM.  The connection, of course, is
subject to your firewall allowing connections from the unidata.ucar.edu domain.
on port 388.

> Is this useful for trying to figure out what we are doing wrong?

'notifyme' is, yes.  The log messages are useful in that they indicate that
'notifyme' is connecting and then being disconnected.

Note that if you see the 'ERROR: nullproc5' and 'NOTE: Exiting' lines _without_
killing your 'notifyme' invocation, then there is some problem with your
setup.

re:
> I have deleted and recreated the ldm queue, and restarted.  So far we
> haven't received any CONDUIT from the NSF web site(i.e., notifyme -h 
> idd.cise-nsf.gov),
> but I am afraid our network person is still tinkering and that may be the 
> reason.

I would verify non-receipt of data by having multiple Xterms running with
different processes in each:

- one with a 'notifyme -vl- -f CONDUIT -h idd.cise-nsf.gov'
- another with a 'notifyme -vl- -f CONDUIT'
- another with an 'ldmadmin watch -f CONDUIT'
- perhaps another with a 'less ~ldm/logs/ldmd.log'

> You
> mentioned several emails back that you could look at your local logs to
> see what and when we are requesting CONDUIT data, or did I misunderstand
> you?

Yes I did that.  I was convinced everything is OK on our side since you
can successfully use 'notifyme' on your machine to see the CONDUIT products
that idd.cise-nsf.gov is receiving.  If the appropriate ALLOW was not in
place on idd.cise-nsf.gov, then your 'notifyme -vl- -f CONDUIT -h 
idd.cise-nsf.gov'
would show that the request was rejected.

> And I did split out the CONDUIT request into its own pqact file, as you
> suggested to help in debugging.

OK.  This will make debugging your one CONDUT action more straightforward.

> One question I had:  on our other grib and grib2 data, we are first
> writing the data to disk via the XCD/LDM filer operating on the data
> coming from our NOAAPORT ingestor system, then ldm is concatenating the
> files via our thredds pqact file.

I would say this a different way:

- one action in a pattern-action file is sending the GRIB/GRIB2 products
  to the XCD binary ingester, ingebin.k, through the 'xcd_run' script

- one or more actions in a pattern-action file is(are) concatenating
  the same GRIB/GRIB2 messages into single output files

The only dependence that one pattern-action entry has on any other is that
it will be executed after the first if it occurs later in a pattern-action
file.  If the actions are in different pattern-action files, the order
that they would be executed is not fixed -- it will be determined by
the Operating System giving timeslices to one 'pqact' invocation or
another.

> Since we are receiving CONDUIT in a
> different way, I assume it has to be written to disk as individual files
> by some means, before the concatenation into a single product file
> occurs?

No.

> I know with XCD there are all those CFG files that handle the
> data processing; what is analagous for the collection of the CONDUIT
> that is not coming to us from a local ldm server (if that makes sense)?

The LDM pattern-action files are the counterpart of the XCD .CFG files.

> And what would be a good time to phone you this AM, so we can perhaps
> get a better handle on the whole process?

I am at home at the moment (I always work from home for a couple/three hours
before heading into Boulder).


Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: WFA-619667
Department: Support IDD
Priority: Normal
Status: Closed