[IDV #JHS-477170]: 20130607: WRF like model aggregation in the IDV

Greetings Chiaying!

In the lower left-hand corner of the IDV is a display of the current time, in
UTC. If you click that clock, the display will change and show you information
about your current memory usage. There are three numbers, seperated by
slashes, and they describe the follow: the first is the amount of memory
currently used by the IDV, the second is the maximum amount of memory used
at any point during your IDV session, and the third is the total amount of
memory allocated to the IDV. Would you mind loading the display you described
and sending me those three numbers? If the second number equals the third,
the IDV can become quite unresponsive.

Also, what is the grid spacing of the WRF output you are viewing?

Thanks!

Sean

> Hi Sean,
> 
> I have another question about loading multiple wrfout files and the IDV
> performance.  While loading 3D data, it takes IDV a noticeable time to
> respond to every commend. Each wrfout I am currently using is about 500 M,
> and I load 36 of them.  I display only the first 15 layers (in total,
> there are 36 layers for 3D fields).  IDV became very slow when I tried to
> display point volume... which took 5 minutes or longer every time when I
> try  to adjust color bar, change view angle, or change vertical range,
> etc.
> we have choice to put data on local machine (MAC) or on the cluster (load
> it trough internet).  I load it through the local machine, which should be
> a more efficient way.
> 
> The system setting I set for IDV is 7000M memory for IDV (I have 8G
> memory), permGen size is 128 M, write grib index in disk Cache, data cache
> Memory Percent 50, unlimited image size, enable geometry by reference, and
> enable access to image data by reference.  I wondering if there is anyway
> I can increase the performance of IDV.
> 
> Also, I sent you and Tom another email about displaying observed
> trajectories from drifter data yesterday, but I forgot to keep the title "
> [IDV #JHS-477170]", and not sure if the email will go to you (and Tom) or
> not.  The automatic response assigned me a number "[IDV #TRI-715395]:
> display drifter trajectory files in IDV".
> 
> Thanks.
> 
> -Chiaying
> 
> 
> > Hi Sean,
> >
> > Thanks for the reply.  I can display/animate all the three model
> > components without problem. Thanks again.
> >
> >
> > -Chiaying
> >> Hi Chiaying!
> >>
> >>> Hi Sean,
> >>>
> >>> Thanks for the answer, I just tried and umwmout.ncml works well now
> >>> after
> >>> the changes!
> >>
> >> Great, I'm glad it worked! See below for the solution to the HYCOM
> >> output.
> >>
> >>>
> >>> I have another question about loading ocean model (HYCOM) outputs (also
> >>> NetCDF file) into IDV and loop them.
> >>>
> >>> In the HYCOM output, there is time-dimension for all the variables. So,
> >>> I
> >>> first tried to load them into IDV with "Aggregate Grids by Time"
> >>> option,
> >>> and then I got an error message saying  "no coordinate variable for agg
> >>> dimension = time."  My first guess was that the dimension name for time
> >>> in
> >>> HYCOM output is "MT" , and IDV might not able to recognize it. another
> >>> possible issue is that MT is written as day since  1900-12-31 (see
> >>> below)
> >>> dimensions:
> >>> MT = UNLIMITED ; // (1 currently)
> >>> Latitude = 1000 ;
> >>> Longitude = 1000 ;
> >>> Depth = 40 ;o
> >>> variables:
> >>> double MT(MT) ;
> >>> MT:long_name = "time" ;
> >>> MT:units = "days since 1900-12-31 00:00:00" ;
> >>> MT:calendar = "gregorian" ;
> >>> MT:axis = "T" ;
> >>> float salinity(MT, Depth, Latitude, Longitude) ;
> >>> salinity:coordinates = "Date" ;
> >>> salinity:standard_name = "sea_water_salinity" ;
> >>> salinity:units = "psu" ;
> >>> salinity:_FillValue = 1.267651e+30f ;
> >>> salinity:valid_range = 9.70682f, 38.86856f ;
> >>> salinity:long_name = " salinity [20.1H]" ;
> >>>
> >>>
> >>> Then, I tried to write a .ncml file to see if I can aggregate HYCOM
> >>> output
> >>> as what you did for umwm. No luck in this either (with an error message
> >>> saying no griddata found for archv.ncml).  Not sure what causes the
> >>> problem.
> >>>
> >>> Can you give me some hint how I should solve this issue?  I put the
> >>> hycom
> >>> data and my archv.ncml online at:
> >>> http://orca.rsmas.miami.edu/~chiaying/IDV/
> >>
> >> Your ncml is very close! The issue is that a time dimension and variable
> >> already
> >> exists, but it's not called "time", which is what the IDV expects when
> >> aggregating
> >> (this is a deficiency in the IDV that needs to be addressed). Since you
> >> already have
> >> time in your files, you can simply do a joinExisting aggregation in
> >> ncml,
> >> rather
> >> than a joinNew, and there is no need to create a new time variable from
> >> the file
> >> name. In order to do this, make the following changes to your ncml file.
> >>
> >> Change the contents of the aggregation tag to:
> >>
> >> aggregation dimName="MT" type="joinExisting"
> >>
> >> Change the contents of the scan tag to:
> >>
> >> scan location="/home/disk/manta9/chiaying/2012_isaac/archvnc/"
> >> suffix=".nc" subdirs="false"
> >>
> >> By the way, the units on the MT variable are perfectly acceptable, so no
> >> issues
> >> there :-)
> >>
> >> If you are interested in learning more about ncml, check out the ncml
> >> tutorial
> >> and cookbook:
> >>
> >> http://www.unidata.ucar.edu/software/netcdf/ncml/v2.2/Tutorial.html
> >>
> >> http://www.unidata.ucar.edu/software/netcdf/ncml/v2.2/Cookbook.html
> >>
> >> Let me know if you have any questions!
> >>
> >> Cheers!
> >>
> >> Sean
> >>
> >>>
> >>> Thanks for your help.
> >>>
> >>> Chiaying
> >>>
> >>>
> >>>
> >>> > Greetings Chiaying!
> >>> >
> >>> > My name is Sean, and I put together the README file and the ncml
> >>> file.
> >>> >
> >>> > First, I would rename the ncml file from umwmout.nml.xml to
> >>> > umwmout.ncml.
> >>> >
> >>> > Then, I would check the following:
> >>> >
> >>> > In the scan tag, check to make sure the dateFormatMark matches
> >>> > how the date is expressed in your file name. For example, the error
> >>> > message you passed along indicates that your files are named like:
> >>> >
> >>> > umwmout_2012-08-26_12:00:00.nc
> >>> >
> >>> > which means the dateFormatMark should be
> >>> >
> >>> > umwmout_#yyyy-MM-dd_HH:mm:ss"
> >>> >
> >>> > The ncml file I sent used a slightly different dateFormatMark.
> >>> >
> >>> > If everything works, then loading the ncml file using "I'm feeling
> >>> > lucky" is all you will need to do (that is, you won't need to do
> >>> > step three as you described in your previous email).
> >>> >
> >>> > Please let me know if it works with these changes.
> >>> >
> >>> > Cheers!
> >>> >
> >>> > Sean
> >>> >
> >>> >
> >>> >> Hi Tom,
> >>> >>
> >>> >> Thanks for the files.  I have tried to use umwmout.nml.xml file.
> >>> >> However,
> >>> >> I got an error message wile trying loading umwmout*.nc data. Here is
> >>> >> what
> >>> >> i did:
> >>> >>
> >>> >> 1) I put umwmout.nml.xml in the same folder as where the unwmout*.nc
> >>> >> are.
> >>> >> 2)  load umwmout.nml.xml with "I'm feeling lucky" data choice (I
> >>> have
> >>> >> changed the path)
> >>> >> 3)  load umwmout*.nc file with "Aggregate Grids by Time" option, and
> >>> I
> >>> >> got
> >>> >> an error message:
> >>> >>
> >>> >> There was an error loading the data:
> >>> >> Error creating data source:aggregated.netcdf.grid with:
> >>> >> [/home/disk/manta9/chiaying/2012_isaac/archvnc/umwmout_2012-08-26_12:00:00.nc,
> >>> >> /home/disk/manta9/chiaying/2012_isaac/archvnc/umwmout_2012-08-26_13:00:00.nc,
> >>> >> /home/disk/manta9/chiaying/2012_isaac/archvnc/umwmout_2012-08-26_14:00:00.nc]
> >>> >> Grid data source failed making data set:
> >>> >> /Users/chiayinglee/.unidata/idv/DefaultIdv/tmp/multigrid_b3ad0ca8-652c-4a8e-bd66-093cd3468f3b0.ncml
> >>> >> AggregationExisting: no coordinate variable for agg
> >>> >> dimension= time
> >>> >>
> >>> >> It seems that  IDV still can not recognize the time.  Am I missing
> >>> >> something? Thanks.
> >>> >>
> >>> >> Chiaying
> >>> >>
> >>> >>
> >>> >>
> >>> >> > Hi Shuyi,
> >>> >> >
> >>> >> > re:
> >>> >> >> Thank you so much for your help on IDV!  My group is learning to
> >>> use
> >>> >> >> IDV for our coupled model output. I really like the feature that
> >>> we
> >>> >> can
> >>> >> >> display/analyze the atmos/ocean/wave fields together in IDV.
> >>> >> >> Milan and Chiaying (CC'd) may email you for some questions as we
> >>> >> >> are learning/experimenting with various functions.
> >>> >> >
> >>> >> > This sounds good.  Did you pass along the README file with
> >>> comments
> >>> on
> >>> >> > how your coupled model output could be better structured for use
> >>> by
> >>> >> > the IDV and other applications that provide support for data in
> >>> netCDF
> >>> >> > format?
> >>> >> >
> >>> >> > Tom
> >>> >>
> >>> >>
> >>> >
> >>> > Ticket Details
> >>> > ===================
> >>> > Ticket ID: JHS-477170
> >>> > Department: Support IDV
> >>> > Priority: Normal
> >>> > Status: Open
> >>> >
> >>>
> >>>
> >>
> >> Ticket Details
> >> ===================
> >> Ticket ID: JHS-477170
> >> Department: Support IDV
> >> Priority: Normal
> >> Status: Open
> >>
> >
> >
> 
> 

Ticket Details
===================
Ticket ID: JHS-477170
Department: Support IDV
Priority: Normal
Status: Open