[THREDDS #TKC-172853]: Missing values and TDS

Hi Cathy:

NaNs are no longer deprecated because all architectures now support them.
We have updated the Unidata doc you found to reflect that.
You have to test for a missing value, im not sure why isnan_ieee() is any more 
trouble than any other missing value.

John


> Ethan
> 
> Yes, As I said, I use something like
> 
> value=1.e20
> > replace_ieeenan (hgt, value, 0)
> > hgt@_FillValue=1.e20
> > hgt@missing_value=1.e20
> 
> where replace_ieeenan is from the routine you suggest.
> 
> When I am reading a lot of data, it's 4 extra lines for all reads.
> 
> Cathy
> 
> address@hidden> wrote:
> 
> > NCL seems to handle NaNs:
> >
> > http://www.ncl.ucar.edu/Document/Functions/Built-in/isnan_ieee.shtml
> >
> > have you tried using that?
> >
> > > Ethan
> > >
> > > My basica problem is that NCL does not interpret NaN correctly. So,
> > before
> > > doing any plotting or analysis, for all reads of CFSR data, I need to
> > have
> > > something like
> > >
> > > replace_ieeenan (hgt, value, 0)
> > > hgt@_FillValue=1.e20
> > > hgt@missing_value=1.e20
> > >
> > > which is extra time and lines of code. This could be interpreted as a NCL
> > > issue of course (it probably is). But, I see in the Unidata doc (from
> > Google
> > > search)
> > >
> > > Using an IEEE infinity or NaN for a fill value is not recommended,
> > because
> > > the resulting file may not be interpreted correctly on platforms that
> > don't
> > > use the IEEE 754 Standard for floating-point. If you use non-normal
> > numbers
> > > for fill values, a fix is to apply this
> > > patch<http://www.unidata.ucar.edu/software/netcdf/patches/vardata-patch>
> > > to
> > > the file ncdump/vardata.c and rebuild *ncdump*. If you don't have the
> > IEEE
> > > finite() function used in this patch in libm, you can try compiling
> > > ncdump/vardata.c with a -DNO_FINITE flag and with compiler flags that
> > > support IEEE comparisons (e.g. NaN != NaN).
> > >
> > > That isn't the issue for NCL but it does indicate that Unidata seems it's
> > a
> > > bad idea to use NaN.
> > >
> > > I haven't checked too much but I know it does work in GrADS and IDV and
> > our
> > > dncdump. I would be very curious about Fortran but don't have compiled
> > code
> > > for OPEnDAP.
> > >
> > > Cathy
> > >
> > >
> > >
> > >
> > > address@hidden> wrote:
> > >
> > > > > Ethan
> > > > >
> > > > > I am using opendap access to the NOMADS server. I open the files in
> > NCL
> > > > > and then read them. I also tested open the files using dncdump
> > > > >
> > > > > dncdump -h
> > > > >
> > > >
> > http://nomads.ncdc.noaa.gov/thredds/dodsC/cfsrmon/197901/ocnh06.gdas.197901.grb2
> > > > >
> > > > > and see NaN as the missing value there. e.g.
> > > > >
> > > > > ......
> > > > > float Vertical_velocity_geometric(time, depth_below_sea, lat, lon) ;
> > > > > Vertical_velocity_geometric:long_name =
> > > > > "Vertical_velocity_geometric @ depth_below_sea" ;
> > > > > Vertical_velocity_geometric:units = "m s-1" ;
> > > > > Vertical_velocity_geometric:missing_value = NaNf ;
> > > > > .......
> > > > >
> > > > > We downloaded the grib file directly and looked at it. It does not
> > have
> > > > > the NaN for any of the values as there is a mask applied when the
> > grib
> > > > > is converted to netcdf (in this case, a land mask).
> > > > >
> > > > > So, it is how the grib files are being converted to netCDF, as you
> > > > suggest.
> > > > >
> > > > > Let me know if I can answer any other questions.
> > > > >
> > > > > Cathy
> > > >
> > > > Hi Cathy:
> > > >
> > > > The rib file does not store the missing values, so each library chooses
> > > > that representation. The netcdf-java library uses NaNs to indicate
> > missing
> > > > values. This currently cannot be changed. What problem are you having
> > with
> > > > portability of NaNs?
> > > >
> > > > Ethan
> > > >
> > >
> > > NCL does not read NaN correctly. So, for all reads, I need to have in
> > > addition
> > >
> > > replace_ieeenan (hgt, value, 0)
> > > hgt@_FillValue=1.e20
> > > hgt@missing_value=1.e20
> > >
> > >
> > > If you use an IEEE infinity as the value of the _FillValue attribute for
> > a
> > > variable, the *ncdump* program will output _ for ordinary values of the
> > > variable, as if they had not been written.
> > >
> > > Using an IEEE infinity or NaN for a fill value is not recommended,
> > because
> > > the resulting file may not be interpreted correctly on platforms that
> > don't
> > > use the IEEE 754 Standard for floating-point. If you use non-normal
> > numbers
> > > for fill values, a fix is to apply this
> > > patch<http://www.unidata.ucar.edu/software/netcdf/patches/vardata-patch>
> > > to
> > > the file ncdump/vardata.c and rebuild *ncdump*. If you don't have the
> > IEEE
> > > finite() function used in this patch in libm, you can try compiling
> > > ncdump/vardata.c with a -DNO_FINITE flag and with compiler flags that
> > > support IEEE comparisons (e.g. NaN != NaN).
> > >
> > >
> > > >
> > > > >
> > > > > On 9/28/11 10:31 AM, Unidata THREDDS Support wrote:
> > > > > > Hi Cathy,
> > > > > >
> > > > > > How are you getting the data? Are you using the NetCDF Subset
> > Service
> > > > (NCSS) access method to request the dataset?
> > > > > >
> > > > > > There is no way to specify that you would like non-NaN missing
> > values
> > > > when you request a netCDF file from the TDS. I suspect it isn't a
> > simple
> > > > matter in this case as it has to do with how the GRIB files are being
> > > > converted into netCDF.
> > > > > >
> > > > > > However, I'm going to ask John Caron to give a more definitive
> > answer.
> > > > Unfortunately he won't be back from vacation for about two weeks.
> > > > > >
> > > > > > Ethan
> > > > > >
> > > > > > Cathy Smith wrote:
> > > > > >> I am reading CFSR data from NOMADS and missing values are being
> > > > returned
> > > > > >> as NaN (for example, for  ocean variables over land) from the TDS.
> > > > This
> > > > > >> causes problems for me in NCL and likely would in other languages.
> > > >  Can
> > > > > >> missing values be returned as something more portable such as
> > > > 9.9692e+36
> > > > > >> as is the default in C?
> > > > > >> Thanks,
> > > > > >> Cathy Smith
> > > > > >
> > > > > > Ticket Details
> > > > > > ===================
> > > > > > Ticket ID: TKC-172853
> > > > > > Department: Support THREDDS
> > > > > > Priority: High
> > > > > > Status: Open
> > > > > >
> > > > >
> > > > > --
> > > > > ----------------------------------------------
> > > > > NOAA/ESRL PSD and CIRES CDC
> > > > > 303-497-6263
> > > > > http://www.esrl.noaa.gov/psd/people/cathy.smith/
> > > > >
> > > > > Emails about data/webpages may get quicker responses from emailing
> > > > > address@hidden
> > > > >
> > > > >
> > > >
> > > >
> > > > Ticket Details
> > > > ===================
> > > > Ticket ID: TKC-172853
> > > > Department: Support THREDDS
> > > > Priority: Critical
> > > > Status: Open
> > > >
> > > >
> > >
> > >
> >
> >
> > Ticket Details
> > ===================
> > Ticket ID: TKC-172853
> > Department: Support THREDDS
> > Priority: Critical
> > Status: Open
> >
> >
> 
> 
> --
> 
> ---------------------------------------------------
> NOAA/ESRL PSD and CIRES CDC
> 303-497-6263http://www.esrl.noaa.gov/psd/people/cathy.smith/
> 
> Emails about data/webpages may get quicker responses from emailing
> 
> address@hidden
> 
> 


Ticket Details
===================
Ticket ID: TKC-172853
Department: Support THREDDS
Priority: Critical
Status: Closed