[netCDFJava #PYP-475421]: Bug in NetCDF-Java 4.3 exposed by this GRIB2 file

On 6/18/2012 9:10 AM, Rich Signell wrote:
> New Client Reply: Bug in NetCDF-Java 4.3 exposed by this GRIB2 file
>
> So as I understand it:
>
> 1) a fix is in the works (or at least on the wish list) so that these
> datasets will be correctly handled in the future
> 2) the NcML you provided me before to fix the problem locally would
> have to be generated for each file of this type, not simply applied to
> an aggregation.
>
> -Rich
>
> On Mon, Jun 18, 2012 at 11:04 AM, Unidata netCDF Java Support
> <address@hidden> wrote:
>> On 6/14/2012 2:54 PM, Rich Signell wrote:
>>> New Client Reply: Bug in NetCDF-Java 4.3 exposed by this GRIB2 file
>>>
>>> Lansing,
>>>
>>> I was expecting you to say that this issue was a bug in the GRIB
>>> reader, and it would get a ticket to be fixed.   But I think you are
>>> saying that the data was written wrong, and therefore the issue will
>>> not be fixed?
>>>
>>> This is one of those "big deal" datasets that it would be great to
>>> read correctly.
>>> http://rda.ucar.edu/pub/cfsr.html
>>>
>>> It's good to know that NcML can fix the issue for a file, but there
>>> are hundreds of these, so I would be better to fix the reader if you
>>> can.
>>>
>>> Thanks,
>>> -Rich
>>>
>>> On Tue, Jun 12, 2012 at 12:53 PM, Unidata netCDF Java Support
>>> <address@hidden> wrote:
>>>>> Lansing,
>>>>>
>>>>> It looks like you are using ToolsUI 4.2, not ToolsUI 4.3.   In toolsUI
>>>>> 4.2, you see one variable, but the times are wrong.   In 4.3, you see
>>>>> two variables with correct times, but we want one variable with
>>>>> correct time!
>>>>>
>>>>> Try this:
>>>>> http://www.unidata.ucar.edu/software/netcdf-java/v4.3/webstart/netCDFtools.jnlp
>>>>>
>>>>> On Mon, Jun 11, 2012 at 1:10 PM, Unidata netCDF Java Support
>>>>> <address@hidden> wrote:
>>>>>>> NetCDF-Java folks:
>>>>>>>
>>>>>>> There appears to be a bug in the GRIB2 decoding problem using
>>>>>>> NetCDF-Java 4.3 on some of the surface pressure data we got from
>>>>>>> http://rda.ucar.edu/pub/cfsr.html
>>>>>>>
>>>>>>> This data is supposed to have a single variable (pressure) with
>>>>>>> uniform hourly sampling, but when we bring it up in NetCDF 4.3 tools,
>>>>>>> we see two variables instead of one, with the 1st having a uniform
>>>>>>> time base of 6 hours (0,6,12), and the 2nd having a non-uniform time
>>>>>>> base that fills in the rest of the hourly data  (1,2,3,4,5
>>>>>>> 7,8,9,10,11  13,14...)
>>>>>>>
>>>>>>> Screenshot: http://screencast.com/t/DWEgKMfy
>>>>>>>
>>>>>>> I put a sample file here:
>>>>>>> http://geoport.whoi.edu/thredds/catalog/usgs/data1/rsignell/test/catalog.html?dataset=usgs/data1/rsignell/test/pressfc.gdas.199801.grb2
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Rich
>>>>>>> --
>>>>>>> Dr. Richard P. Signell   (508) 457-2229
>>>>>>> USGS, 384 Woods Hole Rd.
>>>>>>> Woods Hole, MA 02543-1598
>>>>>>>
>>>>>>>
>>>>>> Hello Rich,
>>>>>>
>>>>>> I'm not sure what's going on, but I tried your file and saw only one 
>>>>>> variable reported.  I've attached a screenshot from NetCDF-Java ToolsUI. 
>>>>>>  Maybe I'm missing something obvious here?
>>>>>>
>>>>>> Regards,
>>>>>>    Lansing Madry
>>>>>>    Unidata
>>>>>>    Boulder, Colorado
>>>>>>
>>>>>> Ticket Details
>>>>>> ===================
>>>>>> Ticket ID: PYP-475421
>>>>>> Department: Support netCDF Java
>>>>>> Priority: Normal
>>>>>> Status: Open
>>>>>
>>>>> --
>>>>> Dr. Richard P. Signell   (508) 457-2229
>>>>> USGS, 384 Woods Hole Rd.
>>>>> Woods Hole, MA 02543-1598
>>>>>
>>>>>
>>>> Rich,
>>>>
>>>> Thanks for educating me on 4.2 vs. 4.3 -- details, details!
>>>>
>>>> I dug a little deeper to see what is going on with the data in the file 
>>>> you sent.  It looks like, at the byte level, there are some differences 
>>>> that 4.2 conveniently ignored and 4.3 notices.  Specifically, if you look 
>>>> at the file in the iosp/GRIB2/GRIB2Collection tab, you'll notice that two 
>>>> variables seem to show up, with separate GDS hash codes.  In that view, if 
>>>> you look down to the Grib2GridDefinitionSection, highlight a row, 
>>>> right-click on the row, and select "Show GDS", you'll see the information 
>>>> for Lo2 differs for the two "different" variables:
>>>>
>>>> Lo2 - longitude of last grid point == 359687000
>>>> Lo2 - longitude of last grid point == 359688000
>>>>
>>>> This is really a defect in how the data was written out - perhaps a 
>>>> rounding error somewhere, or some other loss of precision.
>>>>
>>>> In the meantime, you can "correct" the issue using a bit of NcML to change 
>>>> the GDS hashcode.  Details are here:
>>>>
>>>> http://www.unidata.ucar.edu/software/netcdf-java/formats/GribFiles.html
>>>>
>>>> and I've tried this bit of NcML and found it to work in this case:
>>>>
>>>> <?xml version="1.0" encoding="UTF-8"?>
>>>> <netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2"; 
>>>> location="C:/Users/madry/rsignell/pressfc.gdas.199801.grb2">
>>>>    <iospParam>
>>>>     <gdsHash from="1450192070" to="1450218978"/>
>>>>    </iospParam>
>>>> </netcdf>
>>>>
>>>> However, you will need to delete the associated .ncx file before opening 
>>>> this NcML file in the toolsUI-4.3 viewer.
>>>>
>>>> Let me know if this doesn't work for you, and I'll take another crack at 
>>>> it.
>>>>
>>>> Cheers,
>>>>    Lansing
>>>>
>>>> Ticket Details
>>>> ===================
>>>> Ticket ID: PYP-475421
>>>> Department: Support netCDF Java
>>>> Priority: Normal
>>>> Status: Open
>>>>
>>>
>> Rich,
>>
>> I spoke with someone who has been here a little longer than I have, and
>> he tells me that there is a "fix" that has been previously incorporated
>> into the TDS, but appears not to be in our current version.
>> Essentially, because different machines (or even the same machine, under
>> different circumstances) can write the same data out and not have it
>> look identical past a certain precision, we have to set a cut-off on the
>> precision for hashing purposes.  I apologize for not passing that onto
>> you more clearly, but I've submitted a ticket on it.
>>
>> Regards,
>>    Lansing Madry
>>    Unidata
>>    Boulder, Colorado
>>
>>
>>
>> Ticket Details
>> ===================
>> Ticket ID: PYP-475421
>> Department: Support netCDF Java
>> Priority: Normal
>> Status: Open
>>
>
>
Rich,

I was out Thursday and Friday last week, and responded to an earlier 
e-mail of yours this morning before seeing that John had jumped in 
directly on my behalf  - hope that didn't cause confusion.  My 
understanding (I'll check this) is that if the different grib files have 
even more different hashes of their data, because of different metadata 
associated with each file, then they'll all need to be fixed in a 
similar fashion.  Unfortunately, it seems a bit kludgy and inordinately 
painful, so I'm hoping that we can teach the TDS how to ignore 
unintentional differences in the metadata.  You are correct that it is 
on the wishlist.

Regards,
   Lansing Madry
   Unidata
   Boulder, Colorado



Ticket Details
===================
Ticket ID: PYP-475421
Department: Support netCDF Java
Priority: Normal
Status: Open