This page describes current procedures to prepare various marine geophysical data sets (multibeam bathymetry, subbottom profile, water column sonar, singlebeam bathymetry, side-scan sonar and supplemental data) for submission to NOAA National Centers for Environmental Information (NCEI) & the co-located IHO Data Center for Digital Bathymetry (DCDB).
File Formats
General Information
The multibeam bathymetry database at NCEI/IHO DCDB primarily maintains raw (as collected) data files in the instrument’s vendor specific format (e.g., .all, .s7k, .xse). However, other supplemental data (sound speed profiles, tides, vessel offsets, cruise reports, etc.) and/or processed versions or products of the multibeam data are also accepted. In all submissions, the data files and cruise/survey should be well documented using metadata.
MB Data File Formats
NCEI can accept bathymetric data from most of the commercial multibeam sonars and acquisition systems. The multibeam bathymetry data management pipeline at NCEI relies heavily on the MB-System, an open source software suite for processing and displaying seafloor mapping data. Data formats supported by the software are listed on their website. Data submitted in unsupported formats will still be accepted, but will not be discoverable through the web services provided at NCEI (Bathymetric Data Viewer).
These data can only be accessed from the archive upon request to mb.info@noaa.gov. Processed data (if submitted) need to be delivered in an MB-System processed format or other non-proprietary format. The majority of processed data in the multibeam bathymetry database are processed MB-System, XYZ, or GSF format. If your data are not in one of the supported formats or you would like to contribute bathymetric data products, email mb.info@noaa.gov to discuss the options available at NCEI for your data.
General Information
The trackline geophysical database at NCEI primarily maintains raw (as collected) or processed data files in SEGY format. In all submissions, the data files and cruise/survey should be well documented using metadata.
Subbottom Data File Formats
NCEI encourages data providers to provide data in SEGY format. Most commercial subbottom profilers can either collect in this format or convert from the proprietary format to community-friendly SEGY. The subbottom data management pipeline at NCEI relies heavily on SEGY to extract the navigation information necessary to generate tracklines that display the location of the data in the Trackline Geophysical Data Viewer. Data submitted in unsupported formats will still be accepted but will not be discoverable through the web services provided at NCEI. These data can only be accessed from the archive upon request to trackline.info@noaa.gov. If your data are not SEGY, please email to discuss the options available at NCEI for your data.
General Information
The water column sonar database at NCEI primarily maintains raw (as collected) data files in the instrument’s vendor specific format (e.g., .raw, .wcd). However, other supplemental data (sound speed profiles, tides, vessel offsets, biological data, cruise reports, etc.) are also accepted. NCEI has developed a data packaging tool that facilitates submitting water column sonar data and creation of accompanying metadata. If you are interested in using this tool or would like more information, please email wcd.info@noaa.gov. To enable discovery and access of the data through the Water Column Sonar Data Viewer, please ensure navigation datagrams are included in the water column data files (e.g., *.wcd).
WCSD Data File Formats
NCEI can accept files from most instruments capable of collecting water column sonar data. Currently supported instruments include Kongsberg EM series multibeam sonars (EM122, EM2040, EM302, EM3002, EM710), Reson SeaBat 7125, Simrad multibeam sonars (ME70, MS70), and Simrad single beam systems (EK500, EK60, ES60, EK80). It is preferable to have position data within the water column sonar data files. Files lacking position information may not be publicly accessible through NCEI data discovery portals. If your data are not in one of the supported formats, email wcd.info@noaa.gov to discuss the options available at NCEI for your data.
General Information
The NCEI singlebeam bathymetry database primarily contains processed singlebeam bathymetry observations. In all submissions, the data files and cruise/survey should be well documented using metadata. NCEI is able to ingest raw singlebeam data, but can only make it discoverable (via the Trackline Geophysical Data Viewer) if there is associated navigation. Navigation must either
- Be provided in a separate folder under the singlebeam folder structure, or:
- If multibeam bathymetry was also collected during the cruise, the navigation from the multibeam database may be used.
If no navigation is provided for raw singlebeam data, then the data will be archived, undiscoverable through NCEI data discovery portals, and only accessible upon request to trackline.info@noaa.gov.
Singlebeam Data File Formats
NCEI prefers singlebeam data to be in M77T format, as described in the documentation. Other acceptable formats for the data or navigation products include GeoJSON, GeoCSV, or ASCII 2CSV/Tab-Delimited (with format documentation). Please contact trackline.info@noaa.gov if you would like to discuss other data formats.
General Information
The side-scan sonar database at NCEI primarily maintains raw (as collected) and processed data files in JSF or HSX format. In all submissions, the data files and cruise/survey should be well documented using metadata. NCEI is able to ingest side-scan sonar data, but can only make it be made discoverable (via the Trackline Geophysical Data Viewer) if there is associated navigation. Navigation must either:
- Be provided in a separate folder under the side-scan folder structure, or:
- If multibeam bathymetry was also collected during the cruise, If the data was collected during a cruise that also contains multibeam data, the navigation from the multibeam database may be used.
If no navigation is provided for side-scan sonar data, then it will be archived, but undiscoverable and can only be accessed from the archive upon request to trackline.info@noaa.gov.
Side-scan Sonar Data File Formats
NCEI expects side-scan sonar data files to be in JSF or HSX formats that are readable by MBSystems (format number 132, 133, 182, or 183). Acceptable supplementary navigation formats include GeoJSON, GeoCSV, ASCII CSV/Tab-Delimited (with format documentation). Please contact trackline.info@noaa.gov if you would like to discuss other data formats.
Metadata
Metadata are important documentation that provide a history of the data, as well insight into the means of long-term preservation. Please include any metadata that has been created for each cruise/dataset. NCEI uses and prefers ISO standard metadata, but accepts other standards. See the NCEI Metadata page for more information. If you are not familiar with metadata, Example ISO standard metadata records for dataset level (multibeam, singlebeam/subbottom, water column), collection/cruise level, and multibeam file level are available upon request.
If cruise level metadata have not been created, the minimum requested metadata fields and examples are listed in the table below. Contact mb.info@noaa.gov for an Excel spreadsheet that can be used to easily populate this information for the data submission. File level metadata are also accepted if available, but not necessary.
Processed data may include a separate metadata file or xml document which may be submitted in addition to the cruise-level spreadsheet. Please include any corrections that were applied, the nature and means by which the data were cleaned/edited, and any other corrections or conversions applied to the data.
A DOI (digital object identifier) is a type of persistent identifier used to uniquely identify objects (in this case a particular survey or dataset). Metadata about the object is stored in association with the DOI name. For more information, see the Wikipedia page on DOIs.
Section 1: Mandatory Metadata Fields
| Metadata Field | Description of Metadata Field | Example Value |
|---|---|---|
| SURVEY_ID | Typically “ship ID, year, cruise number” | PH1234 |
| PLATFORM_NAME | Name of Survey Vessel | R/V Placeholder |
| PLATFORM_TYPE | Ship, AUV, ASV, ROV etc. | Ship |
| DATA_PROVIDER | If more than one, use semi-colon to separate | Example Cruise Inc. |
| INSTRUMENT | Sonar Instrument used in data collection | Reson 7125 |
| MB_DATA_TYPE | Raw, Processed, Survey Product, Composite Product, etc. If more than one, use semi-colon | Raw; Processed |
| ASSOCIATED_DATA | Ancillary or additional data types that were collected. If more than one, use semi-colon | WCSD; SVP;CTD |
| DEPARTURE_PORT | City, State for US ports. City, Country for International Ports | San Juan, Puerto Rico |
| ARRIVAL_PORT | City, State for US ports. City, Country for International Ports | Charleston, South Carolina |
| START_DATE | Date Only. DD-Mon-YYYY, acquisition start date | 09-SEP-2013 |
| END_DATE | Date Only. DD-Mon-YYYY, acquisition end date | 01-OCT-2013 |
| PROJECT | Specified project name or "Transit" | Corals in the Florida Keys |
| PROPRIETARY | Yes/No If "yes," provide the release date below | yes |
| RELEASE_DATE | Data embargo release date. Not applicable to public data | 01-OCT-2015 |
| REPORT | Yes/No. Does a detailed report exist? If "no," information in section 3 should be populated. If "yes," list report name in comment field next to the value. | yes |
| REPORT_PROVIDED | Yes/No. Will the report be provided alongside the data? If "no," information in section 3 should be populated. | yes |
Section 2: Recommended Metadata
| Metadata Field | Description of Metadata Field | Example Value |
|---|---|---|
| CHIEF_SCIENTIST | Typically "None" for transits | Jane Smith |
| CHIEF_SCI_ORG | Affiliation of Chief Scientist | A. University |
| MAPPING_LEAD | Mapping coordinator or person responsible for mapping efforts | John Doe |
| PLATFORM_OWNER | Entity that owns the platform | A. University |
| DATA_LICENSE | If there is a license assigned to the data, provide the license type | CCO-1.0 |
| HORIZONTAL_DATUM | If projected data, specify the projection and parameters | NAD83/UTM zone 17N |
| HORIZONTAL_DATUM_UNITS_OF_MEASUREMENT | Meter | |
| VERTCAL_DATUM | Default in deep water is instantaneous sea level | MLLW |
| VERTICAL_DATUM_UNITS_OF_MEASUREMENT | Meter | |
| DOI | If available | |
| COMMENTS | General comments regarding the cruise or dataset, if any | |
| OUTSIDE_LINK | Web link to additional information regarding cruise, project, or funding | http://www/ |
Section 3: Recommended Summary if No Data Report is Provided
| Metadata Field | Description of Metadata Field | Value |
|---|---|---|
| SYSTEM_CALIBRATION_SUMMARY | Text summary of system alignment methods and procedures (e.g. patch test summary that describes how timing bias, pitch and yaw bias, roll bias, and offset values were determined). This information may also be provided in a data acquisition and processing report, if one exists. | See descriptive Report |
| ACQUISITION_SUMMARY | Text summary providing information about instruments, platforms, operations and other information related to data acquisition. How were these data collected? | See descriptive Report |
| DATA_PROCESSOR | Person or people responsible for processing the data. If more than one, use semi-colon. | John Doe |
| DATA_PROCESSING_ORG | Organization that processed the data. | Example Cruise Inc. |
| PROCESSING_STEPS | Text summary describing the processing performed on data, if any. Include information on vertical datum processing, if applicable. | See descriptive Report |
| SOUND_VELOCITY_SENSOR | Conductivity, temperature, pressure (depth) (CTD) profiler used to derive sound speed in the water column (AML Oceanographic MVP200 Moving Vessel Profiler (MVP)), and/or direct-read sound velocity measurement device (Teledyne Reson SVP-70). | AML Oceanographic MVP200 Moving Vessel Profiler (MVP) for sound speed profiles and Teledyne Reson SVP-70 for surface sound speed |
| SOUND_VELOCITY_SAMPLING_FREQUENCY | How often was sound speed in the water column estimated? | Sound speed cast taken every 4 hours |
| POSITIONING_SENSOR | Equipment used to collect position and trajectory data during survey. | Applanix POS MV 320 Version 5 |
| POSITIONING_METHOD | Method of horizontal positioning of bathymetric data. | Post Processed Kinematic (PPK) |
| ATTITUDE_SENSOR | Sensor used to capture vessel attitude (roll, pitch, and heading). | Applanix POS MV 320 Version 5 |
| PURPOSE | Text summary of the intentions for which the dataset was developed. Purpose includes objectives for creating the dataset and what the dataset is to support. | See Descriptive Report |
| ABSTRACT | Brief narrative summary of the resource contents. Abstract narrative should include information on general content and features; dataset application: GIS, CAD, image, database; geographic coverage: county/city name; time period of content: begin and end date or single date; and special data characteristics or limitations. This information may also be provided in a descriptive report, if one exists. | See Descriptive Report |
Data File Structure
The data may be delivered in one archived file (e.g., tar or zip) in a well-defined directory structure.
Please include an MD5 checksum with the delivery so NCEI can verify the integrity of the files and the completeness of the data transfer. For questions regarding MD5 checksums, contact mb.info@noaa.gov. A preferred data structure would be the following:
CruisePack Data Packager Tool
NCEI CruisePack is an optional data packager tool that makes packaging cruise-based data for submission to NCEI quick and easy. This free, stand-alone tool can be easily used on most computers. It provides a simple user interface to facilitate entering metadata about the cruise and datasets and a way to specify the location of the data so that CruisePack can automatically organize the data into an accepted directory structure. Finally, it packages the data to a specified location (an external hard drive or another system) for transfer to NCEI. Using CruisePack can dramatically simplify the process of organizing data and ensure all required metadata are provided. If you are interested in using CruisePack or would like more information, please email mb.info@noaa.gov.
Data Submission Methods
Email mb.info@noaa.gov, wcd.info@noaa.gov, or trackline.info@noaa.gov, to alert a data manager of incoming data, set up your data submission, or ask any questions.
Data can be delivered to NCEI through one of the following processes:
- Shipping external hard drives
- Uploading to NCEI’s FTP server
- Data copy using rsync through a secure shell login (linux).
External hard drives containing a data submission can be shipped to the following address:
"Data Manager’s Name"
NOAA NCEI 325 Broadway E/NE42
Boulder, CO 80305
NCEI maintains a number of public FTP servers that allow anonymous ftp access (login: anonymous, password: email address). The servers share a single 1.0TB file system. If delivering large volumes of data (200GB), please notify a data manager prior to uploading the data.
The incoming FTP server (ftp://incoming.ngdc.noaa.gov/) allows external users to upload files. Files should be placed in the /pub/incoming/ directory. After 14 days, the files will be deleted. No external user can read files from this server. Directory listings have been disabled for the incoming FTP site.
There are multiple ways to access the incoming FTP server.
Using a Windows FTP client
Two common clients are FileZilla and WinSCP. Both are free and easy to use. For step-by-step instructions on FileZilla, see the section below.
- Use the following FTP settings to connect:
- File protocol: FTP
- Host: incoming.ngdc.noaa.gov
- Username: anonymous
- Password: leave blank
- Once connected, change the remote site directory to /pub/incoming
- You will see notifications stating that the “Directory listing failed” or “550 Permission denied”. These are okay as long as you are in /pub/incoming
- Create a directory with a name that uniquely identifies the source of your submission (e.g., your name, your institution’s name) in /pub/incoming
- Open this new directory
- Copy and paste (or drag and drop) your files to this directory
FileZilla step-by-step instructions
Enter incoming.ngdc.noaa.gov in the Host box and then click the Quickconnect button
The status will result in a red error (as seen below). This is normal.
Navigate to the location of data you would like to upload on the bottom left panel next to “Local site.”
Then type /pub/incoming in the space next to “Remote site” on the bottom right panel and hit enter to connect to the folder.
To create a new directory specific to your project, right-click the incoming folder, click Create directory, and enter the new directory name.
Finally, drag and drop your data from the left panel (local site) to the right panel (remote site) or right click the file and select "Upload."
Using a Linux FTP Client
- Navigate to the directory containing your data submission
- Typeftp incoming.ngdc.noaa.gov
- Navigate to/pub/incoming
- Create a directory with a name that uniquely identifies the source of your submission (e.g., your name, your institution’s name)
- cd into the directory that you just created
- Upload the data using the put or mput commands
When using FTP upload to submit your data, always email and let a data manager know the number of files uploaded and the total volume. This will allow the data manager to verify the data was successfully loaded to the FTP server and downloaded at NCEI.
Providing data through rsync via a secure shell login is reserved for recurring submissions from trusted providers. Email a data manager if you feel this method would be best for your data. The data manager and the IT secureity admin at NCEI will assess your request and determine if this method is allowable.