Unidata Users Committee Meeting Summary

18-19 October 2010
Boulder , Colorado

 

Members Attending
Tom Whittaker, Univ of Wisc-Madison-Space Science & Engineering Center, Chair Francina Dominguez, Univ of Arizona Bill Gallus, Iowa State Anne T. Case-Hanks, Univ of Louisiana, Monroe Brendon Hoch, Plymouth State Univ Steve Lazarus, Florida Institute of Technology Patrick Market, Univ of Missouri-Columbia Larry Oolman, Univ of Wyoming Kevin Tyle, Univ of Albany Student Rep: Stefan Cecelski, Univ of Maryland NCEP Reps: Becky Cosgrove (CONDUIT) Michelle Mainelli (GEMPAK-NAWIPS/AWIPS II) USGS Rep: Richard Signell
UPC Staff Attending
Tina Campbell John Caron Julien Chastang Ethan Davis Doug Dirks Ben Domenico Steve Emmerson Ginger Emery Dennis Heimbigner Michael James Yuan Ho Robb Kambic	Brian Kelly Linda Miller Terry Mitchell Mohan Ramamurthy Russ Rew Mike Schmidt Jeff Weber Tom Yoksas

Next Users Committee Meeting 11-12 April 2011

Acronym List

Monday, 18 October 2010

Meeting Agenda 18-19 October 2010

Action Items from 18-19 October meeting

Administrative Items:

• Tom introduced:
  • Francina Dominguez, Univ of Arizona, new committee member
  • Julien Chastang, SW Engineer for IDV and RAMADDA, Unidata
  • Doug Dirks, Scientific Technical Writer/Editor, Unidata
  • Thanks to Bill Gallus, outgoing Users Committee member
• Next Users Committee Meeting will be 11-12 April 2011
• Review of Action Items:
  
  • Michelle Mainelli will provide monthly status updates of NAWIPS/AWIPS II for the Unidata community. They will be posted on the Unidata web site.
  • The lightning data item is discussed below under Lightning item.
  • CONDUIT data is discussed below under the CONDUIT item
  • Site contacts were deferred for this meeting, due to the Unidata survey
  • Stefan is contacting AMS this month
  • The UserComm will send information on GIS (see discussion item)

PolComm Report- Mohan Ramamurthy and Tom Whittaker

The April 15-16 meeting in Arlington had an emphasis on climate. The Deputy Director, Chet Koblinsky, of NOAA's climate services gave an overview of NOAA climate services. Curtis Marshall, NOAA, provided a presentation on the Network of Networks. The Mesonet program has 6M to spend this year and 18M next year.

Director’s Report - Mohan Ramamurthy

• Welcome Francina Dominguez, Univ of Arizona, to the Users Committee. Thanks to Bill Gallus for his service with the Users Committee over the past three years.
• Policy Committee has two new members: Kerstin Lehnert Lamont Doherty Earth Observatory Columbia Univ and Kevin Kloesel, Univ of Oklahoma
• Rotating off of the Policy Committee are Paul Ruscher, Florida State Univ. and Vanda Grubisic, Univ of Vienna. We will be seeing Vanda around NCAR, due to her recent appointment as Director, Earth Observing Lab, NCAR
• Julien Chastang, SW Engineer for IDV and RAMADDA and Doug Dirks, Technical Writer are new staff at Unidata.
• A search is still underway for a technical project manager.
  

This year UCAR and NCAR are celebrating its 50th anniversary. COMET is celebrating their 20th anniversary and Unidata celebrated their 25th anniversary last year. Jeff Weber has coordinated Unidata's participation in the outreach activities associated with the UCAR 50th Anniversary Open House.

Mohan provided some highlights from the status reports.

• RT Data Flows-250+ sites are running LDM-6
• 5.7 TB of data per day are flowing in the IDD
• IDD cluster relays data to more than 650 downstream connections
• THREDDS Data Server (TDS) and Common Data Model (CDM) software- new release earlier this month
• RAMADDA-slowed but there are still users – move to open source and Julien Chastang provides support
• netCDF-2 recent releases
• CF conventions needed for point data-working with GFDL development of GRIDSPEC in lib CF anticipating that this will be part of the CF conventions
• OGC netCDF-CF broader adoption of netCDF in geosciences
• IDV 2.9 in May 2010 release
• GEMPAK surface and upper air data and additional GEMPAK diagnostics
• Display of point cloud data in the IDV-using lidar data
• IDV use at China Meteorological Agency (CMA)
• IDV on upward trend of use-during the last 12 months, the IDV was downloaded 5 times more than GEMPAK and 60 times more than McIDAS
• AWIPS II- LDM is used to file unprocessed  IDD data into AWIPS II scouring hierarchy
• netCDF, netCDF-Java, IDD, UDUNITS
• There were eight equipment awards presented. We will be looking for volunteers for the equipment panel next spring.
• Diversity Activities-DINE college, OU REU students, tribal colleges and Univ of Alaska distance education professors visited Unidata
• Jeff Weber was SOARS mentor to Vanessa Vincente. They created a case study of Hurricane Wilma. Vanessa will be at the AMS Annual Meeting to present a paper on the project.
• <V3.unidata.ucar.edu> is Unidata's new web site-please send comments to Jennifer Oxelson (oxelson@unidata.ucar.edu). It has not been open to the public yet.
• Training workshop-a total of 78 registrants attending from eight countries
  Last year 91 attendees, due to large numbers of netCDF registrations
• The Unidata Community survey conducted by the Users Committee yielded 240 responses who started the survey and 214 who completed the survey
  • In 2001-148 people completed the survey
  • Overall, the community seems happy
  • Research use is higher than it was in previous surveys
  • There is broad international participation
  • GEMPAK concerns were voiced
  • Status of AWIPS II requires more communication; community is anxious about the transition from GEMPAK to AWIPS II
  • Users are not aware of IDV capabilities
  • GIS needs to be explored by Unidata

Discussion:
There was a question pertaining to Unidata's budget and status. Mohan said that all areas are green; exception staffing.
Community relations - GEMPAK transition to AWIPS II needs additional emphasis and updates. Discussion below rendered an action item for Michelle Mainelli to provide a monthly status report for the community.

The budget has been tight, but Unidata is doing better now due to funding for the coming year which was received earlier than usual from NSF. Planning for coming years is difficult. Each government agency has been asked to figure out a budget with cuts by 5%. Each program should plan for this scenario, in the event that it comes to fruition. The NOAA Climate Service has been approved, but no budget has been provided for them to go forward. A Continuing Resoluton (CR) for FY 2011 appropriations may go into affect for the entire year.

The question of survey responses was discussed briefly:

• Must drill down to comments - at a glance, data services, data streams, enhancements to software, and climate are important areas for Unidata. Unidata tools and data are already in use, however, educators seem to be taking advantage of the Internet and Web applications. It appears that they do not need advanced tools like IDV, GEMPAK, AWIPS II, getting data from Unidata and using the tools for education as much as researchers need the tools. There will be more careful analysis of the survey results by the Users Committee. This has not happened yet.

Status Reports:

• Case Study developed by Vanessa Vincente, with Jeff Weber's help, will be presented at AMS by Vanessa.
• COMET has not yet integrated Unidata technologies; collaboration is ongoing but slow-see advantages of using  bundles, etc Jeff is communicating with Wendy at COMET so there is mutual interest.
  
• Definition of case study includes data, ancillary information, news articles, digital information. Others can contribute to the case studies-see more participation from users. Encourage people on IDV mailing list if they have an event - it’s a community effort
• Rich has created information that could be used for an ocean case study

ACTION 1. Doug Dirks will collaborate with Rich Signell on case studies and ocean use of Unidata tools for the E-letter.

• Doug Dirks is requesting input for the e-letter. He will be providing the stats about newsletter interest, so we will have metrics to follow and gauge the readership from now on.
• If Usercomm has ides for the themes Equipment Awards, please let Terry Mitchell know.
• 78 registrants for Unidata workshops taking place in Oct-Nov 2010
• Note that it is important to make people aware of standards and to adhere to them
• GEMPAK final version 6.2 release will be Michael's last major release
• 18 months after AWIPS II- community can take over support for GEMPAK
• Question of updates provided for GEMPAK - NCEP could implement if useful to NCEP users. Scott Jacobs, NCEP, is the focal point.
  IDV status report:
  • Publication produced postscript in PDF
    Extensive use of graphics card
    Publication quality-VisAD package
    Vectors are the issue-irregular contour lines, adjust the fonts with labels-GEMPAK contour fonts are still lacking contouring lines and labels
    Hidden information behind the drop down menu-vis trails
• International activities-data can WMO Resolution 40, but it cannot go into government or companies
• LDM 6.9 will be released soon
• NOAAPort package needs some changes, but will be done soon
• NWS moving toward tripling bandwidth-underlying UCAR network is 10Gb we can triple that.
  • What can the end user handle?
  • Data push-critical data sets
  • Other data –THREDDS, RAMADDA
  • We will need to choose relays ¼ degree GFS—CONDUIT—level 2
    netCDF-libCF - if it works well  then would go through CF convention-climate models-unstructured grids supported by netCDF Java.  Needs coordination-GridSpec mailing lessons-multi file data set
  • Source code availability would be easier to understand why Support isn’t responding, there would be a level of transparency
  • If bug tracking was available to developers, they would get contributions back
    

ACTION 2: UPC will continue to explore tools for tracking of support, bugs, etc. to allow visibility by the user community.

Discussion:
Barbara Ransom, NSF visitor to Unidata, suggested that it is important to have university collaborators who are familiar with the project/program on panels at NSF. Upon proposal submission, give a list of names for the panel to Barbara. NSF reviews are written by panels.  Proposal PI's should always talk to the program officer directly before sending the proposal.  You know who your audience is. Remember that the Panel does not decide, but they provide the information to NSF to contribute toward the decision.

AWIPS II (same presentation for UserComm and PolComm) - Michelle Mainelli

Michelle covered the current status of the migration from GEMPAK/NAWIPS to AWIPS II

• Schedule - AWIPS II software distributed to Unidata community winter 2011/2012
• Over 90% software migration is complete
• The software is designed for backward capability
• Migration code to Raytheon by Nov/Dec
• NSHARP-migrated NAWIPS GUIs/utilities
• Last couple months-finalized hardware configuration
• Will be able to handle both 32 and 64 bit configuration
• All centers have AWIPS II systems along with the additional test bed at
• URL that might be of interest: http://mag.ncep.noaa.gov (NCEP Central Operations)
• Test bed will be ready by late Dec 2010
• NCEP data plug-ins decoders – 29 Oct – implementation Dec 2010
• Transition time Jan 2011-June 2012
• Consolidation – summer 2011- FY12
• Open test bed for the Unidata community
• Weekly migration telcon – training FY11, Q2 and FY12
• Software through the AWIPS program center
• Weather Event Simulator-development 2012/delivery 2013
• GEMPAK ensemble in AWIPS II improved
• Decoding of data will occur in EDEX
• NCEP will provide monthly updates
• GEMPAK will be supported by Unidata for 18 months after first release, currently scheduled for summer 2013
• Licensing of software
  • GEMPAK was developed by Federal Government – pen software
  • Once implemented AWIPS II will have open software policies
    

There is a new GIS project at NCEP to create one stop shop to access GIS/format data in KML, shape files create a place to access GIS format data. Whatever data format they use will be OGC compliant. Suggestions welcome.

Action 3:  Michelle Mainelli will provide monthly status updates from NWS/NCEP to post on the web site to keep the Unidata community apprised of the latest activities regarding GEMPAK/NAWIPS migration

There was a question regarding training on AWIPS II. A week-long training course will take place for NWS sysadmin. Michael and others from Unidata will be invited. The test bed will also be open to Unidata for training purposes.

Discussion: The idea is to have two instances of EDEX; one for case studies and one real time. Decoders will be written to the EDEX data base. Will not have cross section capability immediately, but will eventually be added.

Michael provided a demo. D2D is not installed. NMAP perspective is what is used by Michael. The HDF 5 content is not necessarily complete. It will offer a good way to do case studies, but will not be ready on day one. There is no KML capability yet.

CONDUIT - Becky Cosgrove

Highlights from Becky's presentation include:

There are 58 GB of data per day flowing through CONDUIT. NCEP has upgraded the LDM and GribInsert utility on CONDUIT. There are three CONDUIT servers at NOAA/Space Weather Center in Boulder. The servers feed NOMADS for model data.
Some recent additions are:

• NDFD forecast grids – server at the WOC then to CONDUIT
• Quarter degree output of 0.25 is out to 28 km-needs computer resources
• RTOFS needs to be added

Available data - RUC 18 hr is available. EMC has information posted on the models and listings. We need to send an email to the community list soliciting what they want, and what they are willing to give up. The systems at the WOC need an upgrade for more memory to comply with more data needs. The Climate Forecast System will be available January 2011.

ACTION 4: Becky will provide the descriptions, what’s available and will include URLs for information and sample products.

• The UPC will send a note to the community; provide model descriptions, types available, ask what the community wants...and what they can give up
  

Blue Sky Session: Tom Whittaker, all

This session is meant to provide input to the Unidata technical staff on future ideas and dreams from the committee members. Here's a list:

• Unidata app for I-phone - suggestion is to run Unidata on the I-phone, IDV with automatic updates
• From a teaching perspective-IDV fictional air parcel move up and down compare dry and moist – 3-D geo reference display-visual change
• Presentation graphics in IDV---currently gets lost in the data-publication graphics (labels, contour lines need work) look into multiple plots-share the comparison
• WRF run, 2 km WRF simulations-read into AWIPS II and have ability to print out
• Watershed boundaries-have GIS features and compare with flow data
• netCDF –may need different type of display, may work with shape file
• GIS – wind energy high resolution grids-GIS format
• Airborne remote sensing-CF radial-one common format for all instruments
• Visualization lab -3D visualization
• Wind fields and projecting trajectories
• Sounding data-visualize the sound-basic teaching tool
• Parcel attributes could provide Hovmoller diagrams
• Use the Developers blog
• Reconvene IDV steering committee
• McIDAS V and IDV – discuss the synergies
• Climate data-output paleo climate data, climate SST, repository, IDV and RAMADDA for a repository
• More emphasis-CDM-interoperability – common data tool set
• Matlab, python, Netcdf java sits on top of Java IDL –libcf if CDM was available
• Special session with Python at AMS annual meeting
• CF conventions for NCEP and NWS
• International operability bandwagon-WMO meetings-team on table driven codes-one version of each parameter – takes 12-18 months for acceptability at WMO
• Need standard data BUFR
• Meteorograms, need to be added to AWIPS II
• GIS data sets is used from NWS, data sets used should let Michelle know
  

Demo of next generation LDM - Steve Emmerson

Steve provided a brief demo of where the next LDM is going. Currently, it is being designed as a publish/subscribe model, organizing the data into files and breaking the files into pieces. It will have Bit Torrent like dynamic routing capabilities and is being developed in Java.

Action 5:   UserComm members will send sample GEMPAK scripts to a RAMADDA repository  to aid Unidata in identifying what is missing with IDV. If not sufficient, then solicit input from the gembud list.

Yuan should follow up with CMA person to see how he creates all of the PNG image files in 6 minutes.

Exposing workflow from IDV would be useful.

Survey Discussion: All

The results need to be distributed to the community, but they need to be analyzed thoroughly first.

The next steps are to consolidate the results and send the information to the community via E-letter.

Several approaches were discussed; even the idea of resending the survey to gather additional information. The decision was made to forego additional steps to making the survey available again, and to use the information received to date.

Comments (without personal attribution) need to be included to allow the community to see what others are thinking, and to view the general trends and perceptions within the community. The information needs to be organized and published in the newsletter, along with the next steps that will be taken by the UPC. There are many changes noted since the last survey nearly 10 years ago. One of these is the high number of users who depicted themselves as researchers over educators.

Ideas were shared among the committee and staff on next steps. Some of the points made were:

• Many students to to Web sites for data and tools. It's a sign of the times.
• Need to nurture the students who are interested in the applications, ie. IDV
• Educators need to teach-have the bundle next to the image for IDV-need to make things easier for the users
• GEMPAK scripts need to be put into an area for common use
• Have FAQ's on the Web site

Communication needs to be enhanced on all of these issues.

Lightning Data – Mohan, et al

NLDN - SUNY-Albany (Univ of Albany)-Vaisala data has been flowing to about 60 universities without problems for a long time. The data stopped flowing on Sept 1. There were technical changes made at Vaisala which required changes at SUNY-Albany. Dave Knight and Tom Yoksas are working through these changes which are still underway, but will be completed soon.

USPLN-Chris Herbster stepped forward to facilitate the connection (another point-to-point connection) to the USPLN. We are hoping that Chris will continue to be the focal point for the global USPLN data for the community.

USPLN data is flowing-WSI to see if universities can gain access to global-one hour delayed data.  Both data streams are only for research and education and no rebroadcasting is permitted.

Numerous action items stemmed from the general discussion. They are:

Action 6:  Tom Whittaker will draft a letter to WSI indicating that it is preferred to have the global data in real time, rather than with a 60 minute delay.  This will be brought to the Policy Committee and requested that it be co-signed.
Action 7:  Doug Dirks will provide an E-letter article about the lightning data.
Action 8:  DeSouza Award solicitation should provide the template for nominations. 
Action 9:  Anne Case-Hanks, Kevin Tyle, and Patrick Market are going to go through the survey comments, group them for  the UPC so they can provide responses (grouping to be provided to UPC by 1 November)
Action 10:  Tom Whittaker will draft a letter to David Knight thanking him for working through the issues related to NLDN data distribution, and the past 14 years of data provision.
Action 11:  Send Michelle Mainelli an email about what GIS data (via a web interface) they would like to get from NWS, and in what format.
Action 12: Tom Whittaker will work with Jennifer Oxelson to get the graphs and text from the survey to send to the community.

Linda Miller
Community Services - Unidata
University Corporation for Atmospheric Research
P.O. Box 3000
Boulder, CO 80307-3000
303 497-8646 fax: 303 497-8690