Re: NetCDF and SRB

Hi Arun:

Before i try to answer any of your questions, I have a couple of my own:

1. Does SRB have a "data model" of some sort? I was under the impression that it mostly moved files around. NetCDF, in contrast, allows clients to extract subsets of a file's contents. 

2. What functionality does it provide in its data access protocol? Do you have 
a brief summary that i could look at?


Arun Jagatheesan wrote:
> John:
> I have been requested to formulate a technical strategy for  NetCDF and SRB.
> The expected outcome is an interface or integration at the API or protocol
> level or it could be an architecture using which users can take advantage of
> both SRB and NetCDF. I don't really know the commitment or requirement on
> this effort at this stage. However, I think it worth to get this started
> (this has been on our list for a long time).
>
> SRB is a data grid management system for distributed management of data. It
> does not have its own data format. However it has it own protocol for data
> access. I just started reading some information from NetCDF webpages and
> also the java guide:
> ftp://ftp.unidata.ucar.edu/pub/netcdf-java/v2.2.14/NetcdfJavaUserManual-2.2.
> doc
>
>
> Based on my current level of understanding (which is just based on browsing
> NetCDF webpages), it seems I could use a two-phase approach:
> Phase 1: Initial, get it started phase (developer friendly  - short term)
> Phase 2: More detailed infrastructure level (user friendly - long term)
>
> In phase-1 I could add a SRB-IOServiceProvider to NJ22. This will increase
> our knowledge of NetCDF also. The end result is an added package for NJ22 -
> This might require me having access to your CVS (if you prefer me committing
> this to your package - or I will have to have it on our side).
>
> In phase-2, I could read about the common data model and try to see how it
> could be incorporated for SRB objects.  I really don't know much here and
> need to read more or get a knowledge dump from anyone from your team working
> on CDM. We already support some small HDF function calls in SRB. The idea
> here is that the user could access NetCDF (or CDM files) that are
> distributed in multiple sites as though they are local. SRB would manage
> global namespace of all the distributed files, search based on meta data,
> perform data archival and other data management functions. NetCDF could
> provide the formatting or syntactic layer required to access data. We have
> done some thing similar for HDF5 before for a subset of HDF functions. We
> could do something similar for NetCDF or CDM (which ever would be
> appropriate).
>
> Let me know if you have any suggestions on how I could proceed. I would also
> require CVS access (at a lager stage) to NJ22 if you prefer me committing
> the SRB IOProvider as another package. 
>
> Thanks,
> Arun
> ~~~~~~~~~
> Luck is what happens when preparation meets opportunity.
>
> Arun Jagatheesan
> http://www.sdsc.edu/~arun/
> San Diego Supercomputer Center.
> (858)822.5452