[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[netCDFJava #BNA-191717]: chunking in Java



Hi Jeff,

How chunking and compression affect file size and read/write performance is a 
complex issue. I'm going to pass this along to our chunking expert (Russ Rew) 
who, I believe, is back in the office on Monday and should be able to provide 
you with some better advise than I can give.

In the mean time, here's an email he wrote in response to a conversation on the 
effect of chunking on performance that might be useful:

http://www.unidata.ucar.edu/mailing_lists/archives/netcdfgroup/2013/msg00498.html

Sorry I don't have a better answer for you.

Ethan 

Jeff Johnson wrote:
> Ethan-
> 
> I made the changes you suggested with the following result:
> 
> 10000 records, 8 bytes / record = 80000 bytes raw data
> 
> original program (NetCDF4, no chunking): 537880 bytes (6.7x)
> file size with chunk size of 2000 = 457852 bytes (5.7x)
> 
> So a little better, but still not good. I then tried different chunk sizes
> of 10000, 5000, 200, and even 1, which I would've thought would give me the
> original size, but all gave the same resulting file size of 457852.
> 
> Finally, I tried writing more records to see if it's just a symptom of a
> small data set. With 1M records:
> 
> 8MB raw data, chunk size = 2000
> 45.4MB file (5.7x)
> 
> This is starting to seem like a lost cause given our small data records.
> I'm wondering if you have information I could use to go back to the archive
> group and try to convince them to use NetCDF3 instead.
> 
> jeff


Ticket Details
===================
Ticket ID: BNA-191717
Department: Support netCDF
Priority: Normal
Status: Open


pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy