Add the CDF specs

Nathanael Jourdane
1 parent 5ab287f2
Showing 2 changed files with 471 additions and 30 deletions Show diff stats
CDF_spec.md
readme.md
@@ -0,0 +1,420 @@
+# Formating CDF files for AMDADB
+
+This specification follows the ISTP guidelines, see [the ISTP guide](https://spdf.gsfc.nasa.gov/istp_guide/).
+
+---
+
+## Global CDF attributes
+
+See [the ISTP guidelines for global attributes](https://spdf.gsfc.nasa.gov/istp_guide/gattributes.html).
+
+### Acknowledgement
+
+Value of node `NumericalData/ResourceHeader/Acknowledgement` from SPASE file *NumericalData* corresponding to the granule.
+
+> **Ex:**
+>
+>     NASA, Prof. Henri Reme, European Space Agency
+
+### Data_type
+
+Dataset name
+
+> **Ex:**
+>
+>     PRP
+
+### Data_version
+
+    1
+
+### Descriptor
+
+Instrument name
+
+> **Ex:**
+>
+>     CIS
+
+### Discipline
+
+- If target name is `sun` or `earth` : ***Space Physics***
+- otherwise, look for node value of `NumericalData/ObservedRegion` from the *NumericalData* SPASE file corresponding to the granule:
+	- if it's `Magnetosphere`: ***Space Physics>Magnetospheric Science***;
+	- if it's `NearSurface`: ***Space Physics>Ionospheric Science***;
+	- if it's `Heliosphere`: ***Space Physics>Interplanetary Studies***.
+
+### Generated_by
+
+    CDPP>Centre de Donnees de la Physique des Plasmas
+
+### Generation_date
+
+Value of node `NumericalData/ResourceHeader/ReleaseDate` from the *NumericalData* SPASE file corresponding to the granule, formated as *yyyymmdd*.
+
+> **Ex:**
+>
+>     20151019
+
+### HTTP_LINK
+
+    http://amda.cdpp.eu
+
+### Instrument_type
+
+Value of node `Instrument/InstrumentType` from the *Instrument* SPASE file corresponding to the granule.
+
+> **Ex:**
+>
+>     ElectrostaticAnalyser
+
+### LINK_TEXT
+
+Value of node `NumericalData/ResourceHeader/AlternateName` from the *NumericalData* SPASE file corresponding to the granule **+** ` data are available at `.
+
+> **Ex:**
+>
+>     Cluster 1 Prime Parameter CIS data data are available at
+
+### LINK_TITLE
+
+    CDPP AMDA DATABASE
+
+### Logical_file_id
+
+Dataset name **+** `_` **+** `start_time` epn-core value, formated as `YYYYMMDDHHMMSS`.
+
+Note: This is actually the file name without extension.
+
+> **Ex**
+>
+>     CLUST1_CIS_PRP_19900302040000
+
+### Logical_source
+
+Dataset name
+
+> **Ex:**
+>
+>     CLUST1_CIS_PRP
+
+### Logical_source_description
+
+Value of node `NumericalData/ResourceHeader/Description` from the *NumericalData* SPASE file corresponding to the granule.
+
+> **Ex:**
+>
+>     This instrument (CIS: Cluster Ion Spectrometry) is capable of obtaining full 3D ion distributions with high time resolution (in one spacecraft spin) and mass-per-charge resolution. The experiment consists of [...]
+
+### Mission_group
+
+- Last element of the value of node `Observatory/ObservatoryGroupID` from the *Observatory* SPASE file corresponding to the granule.
+- if the node doesn't exist: value of node `Observatory/ResourceName`.
+
+> **Ex:**
+>
+>     Cluster
+
+### MODS
+
+    Extracted from NetCDF AMDA internal data.
+
+### Parents
+
+The NetCDF file name.
+
+> **Ex**
+>
+>     CLUST1_CIS_PRP_19900302040000.nc
+
+### PI_affiliation
+
+Value of node `Person/OrganisationName` from the *Person* SPASE file corresponding to the granule.
+
+> **Ex**
+>
+>     ESTEC, ESA
+
+### PI_name
+
+Value of node `NumericalData/Contact/PersonID` where `Role` node value is `PrincipalInvestigator`, from the *NumericalData* SPASE file corresponding to the granule.
+
+> **Ex:**
+>
+>     Henri.Reme
+
+### Project
+
+A list with:
+
+- `CDPP>Centre de Donnees de la Physique des Plasmas`
+- the value of node `NumericalData/ProviderName`, from the *NumericalData* SPASE file corresponding to the granule.
+
+> **Ex:**
+>
+>     - CDPP>Centre de Donnees de la Physique des Plasmas
+>     - CSDS-CNES
+
+### Rules_of_use
+
+    System Use Policy
+
+    Thank you for acknowledging the use of AMDA in publications with wording like "Data analysis was performed with the AMDA science analysis system provided by the Centre de Données de la Physique des Plasmas (CDPP) supported by CNRS, CNES, Observatoire de Paris and Université Paul Sabatier, Toulouse."
+
+	Data use policy from originating <NAME> data center: <URL>
+
+where :
+- <NAME> is the value of node `NumericalData/ProviderName` from the SPASE file *NumericalData* corresponding to the granule, or CDPP if the value doesn't exist.
+- and <URL> is kept from this dictionary (according to <NAME>).
+
+- CDPP: http://amda.irap.omp.eu/help/licenceAction.do.html
+- CDAWEB: http://spdf.gsfc.nasa.gov/new_users.html
+- PDS: http://ppi.pds.nasa.gov/citations_policy.jsp
+- MAPSKP: http://mapskp.cesr.fr/use_policy
+- THEMIS: http://themis.ssl.berkeley.edu/roadrules.shtml
+- MAVEN: http://lasp.colorado.edu/home/maven/files/2014/09/MAVEN_public-rules-of-the-road-v3.pdf
+- CASSINI/RPWS/SKR: http://amda.irap.omp.eu/help/SKR.html
+- VEXGRAZ: http://amda.irap.omp.eu/help/VEXGRAZ.html
+- CAA: http://caa.estec.esa.int/caa/ror.xml
+- CSA: http://www.cosmos.esa.int/web/csa/rules-of-the-road
+- STEREO: http://amda.irap.omp.eu/help/STEREO.html
+- CCMC: http://ccmc.gsfc.nasa.gov/PubPolicy.php
+
+
+> **Ex:**
+>
+>     System Use Policy
+>     
+>     Thank you for acknowledging the use of AMDA in publications with wording like "Data analysis was performed with the AMDA science analysis system provided by the Centre de Données de la Physique des Plasmas (CDPP) supported by CNRS, CNES, Observatoire de Paris and Université Paul Sabatier, Toulouse."
+>     
+>     Data use policy from originating CDPP data center: http://amda.irap.omp.eu/help/licenceAction.do.html
+
+### Software_version
+
+Version of the NetCDF to CDF converter.
+
+### Source_name
+
+The mission name
+
+> **Ex:**
+>
+>     clust1
+
+### TEXT
+
+The value of node `NumericalData/ResourceHeader/InformationURL/Name` **+** `, ` **+** the value of node `NumericalData/ResourceHeader/InformationURL/URL`, from the SPASE file *NumericalData* corresponding to the granule.
+
+> **Ex:**
+>
+>     NSSDC Master Catalog listing for Cluster II Rumba Cluster Ion Spectrometry (CIS), http://nssdc.gsfc.nasa.gov/nmc/experimentDisplay.do?id=2000-045A-02
+
+### Time_resolution
+
+`time_sampling_step` value of epn-core, in seconds + ` seconds`.
+
+> **Ex:**
+>
+>     3 seconds
+
+### TITLE
+
+The value of node `NumericalData/ResourceHeader/AlternateName`, from the SPASE file *NumericalData* corresponding to the granule.
+
+> **Ex:**
+>
+>     Cluster 1 Prime Parameter CIS data
+
+### PDS_Start_time
+
+`time_min` value of epn-core, in ISO time.
+
+> **Ex:**
+>
+>     2007-02-13T09:08:43Z
+
+### PDS_Stop_time
+
+`time_max` value of epn-core, in ISO time.
+
+> **Ex:**
+>
+>     2007-02-14T09:08:43Z
+
+### PDS_Observation_target
+
+`target_name` value of epn-core. Use a list of values if necessary (split the `#` character).
+
+> **Ex:**
+>
+>     Saturn
+
+### Observation_type
+
+Look for the child node in `NumericalData/Parameter` node, from the SPASE file *NumericalData* corresponding to the granule.
+
+It is one of: `Field`, `Particules`, `Wave`, `Mixed`, `Support`.
+
+> **Ex:**
+>
+>     Particle
+
+### Other epn-core parameters
+
+Add other epn-core parameters, with `VESPA_` prefix, only if:
+- the value is not empty;
+- this is not the same value as the other element of a tuple (ie min/max).
+
+> **Ex:**
+>
+>     - VESPA_spase_measurement_type: EnergeticParticles
+>     - VESPA_spatial_frame_type: cartesian
+
+---
+
+## Variable attributes (TBC)
+
+See [the ISTP guidelines for variables](https://spdf.gsfc.nasa.gov/istp_guide/variables.html).
+
+To create new variables with [spacepy.pycdf)(http://pythonhosted.org/SpacePy/pycdf.html), use [cdf.new()](http://pythonhosted.org/SpacePy/autosummary/spacepy.pycdf.CDF.html#spacepy.pycdf.CDF.new) and not `attr[]`.
+
+To name variables, use the `NumericalData/Parameter/ParameterKey` parameter from the SPASE file.
+
+### `CATDESC`
+
+type: CDF_CHAR
+
+The value of node `NumericalData/Parameter/Description`.
+
+> **Ex**
+>
+>     Flux density spectrogram measured on the RH polarized array.
+
+### DEPEND_0
+
+type : CDF_CHAR
+
+    Epoch
+
+### LABL_PTR_1
+
+Note: Only if the node value of `NumericalData/Parameter/RenderingHints` is `time series` and the node `NumericalData/Parameter/Structure` exist.
+
+Note 2: `LABL_PTR_1` and `DEPEND_1` can not are both presents.
+
+**Step 1:** Get the labels, which are the nodes value of `NumericalData/Parameter/Structure/Element/Name`.
+
+> **Ex (for skr151150000.nc.gz):**
+>
+>     - RH 100-400kHz
+>     - LH 100-400kHz
+>     - RH 10-1000kHz
+>     - RH 10-1000kHz
+
+**Step 2:** Create a new variable stored with these labels. Then `LABL_PTR_1` is the ID of this variable.
+
+### DEPEND_1
+
+Note: Only if the node value of `NumericalData/Parameter/RenderingHints` is `spectrogram`.
+
+Note 2: `LABL_PTR_1` and `DEPEND_1` can not are both presents.
+
+type : CDF_CHAR
+
+From the file `<dataset_name>_info` from the dataset directory, look for `energy` or `frequency` variables value.
+
+### DICT_KEY
+
+type: CDF_CHAR
+
+> **Ex:**
+>
+>     electric_field>power
+
+### DISPLAY_TYPE
+
+type: CDF_CHAR
+
+> **Ex:**
+>
+>     time_series
+
+### FIELDNAM
+
+type: CDF_CHAR
+
+> **Ex:**
+>
+>     RR
+
+### FILLVAL
+
+type: CDF_UINT1
+
+> **Ex:**
+>
+>     255
+
+### LABLAXIS
+
+type: CDF_CHAR
+
+> **Ex:**
+>
+>     RH flux density
+
+### UNITS
+
+type: CDF_CHAR
+
+> **Ex:**
+>
+>     dB
+
+### VALIDMIN
+
+type: CDF_UINT1
+
+> **Ex:**
+>
+>     0
+
+### VALIDMAX
+
+type: CDF_UINT1
+
+> **Ex:**
+>
+>     255
+
+### VAR_TYPE
+
+type: CDF_CHAR
+
+> **Ex:**
+>
+>     data
+
+### SCALETYP
+
+type: CDF_CHAR
+
+> **Ex:**
+>
+>     lin
+
+### SCALEMIN
+
+type: CDF_UINT1
+
+> **Ex:**
+>
+>     0
+
+### SCALEMAX
+
+> **Ex:**
+>
+>     255
+
+### UCD
 # CDF tools
  
-## NetCDF to CDF converter
+## NetCDF to CDF converter for AMDADB
  
 - File: [nc2cdf.py](./nc2cdf.py)
 - Python interpreter: 3.6
@@ -11,87 +11,108 @@
  
 Convert the NetCDF file, save it in a temp directory, then display its path:
  
-    ./nc2cdf.py path/to/input_file.nc.gz
+```bash
+./nc2cdf.py path/to/input_file.nc.gz
+```
  
 Convert a Net-CDF file and save it in the specified path.
  
-    ./nc2cdf.py path/to/input_file.nc.gz path/to/output_file.cdf
+```bash
+./nc2cdf.py path/to/input_file.nc.gz path/to/output_file.cdf
+```
  
-**Note:** If the specified input file is a gzip archive, it will be automatically extracted before the conversion.
+**Note:** If the specified input file is a gzip archive, it will be automatically extracted in a temp directory before the conversion.
  
 #### Describing a NetCDf file:
  
-    ./nc2cdf.py  -i path/to/file.nc.gz
+```bash
+./nc2cdf.py  -i path/to/file.nc.gz
+```
  
 This display information about a Net-CDF file (such as global attributes and variables information).
  
-### Python library usage
+### Python usage
  
-    import nc2cdf
+```python
+import nc2cdf
  
-    netcdf = NetCdf('path/to/input_file.nc.gz')
+netcdf = NetCdf('path/to/input_file.nc.gz')
  
-    netcdf.describe()
+netcdf.describe()
  
-    netcdf.get_cdf()
-    print('CDF path: ' + netcdf.get_cdf_path())
+netcdf.get_cdf()
+print('CDF path: ' + netcdf.get_cdf_path())
  
-	netcdf.get_cdf('path/to/output_file.cdf')
+netcdf.get_cdf('path/to/output_file.cdf')
+```
  
-## About NetCDF4 and pycdf
+## Dependencies
  
-### NetCDF4
+- NetCDF4
  
 [NetCDF](https://www.unidata.ucar.edu/software/netcdf/) is C library to read and edit NetCDF files.
  
-[NetCDF4](https://github.com/Unidata/netcdf4-python) is a Python wrapper for NetCDF, which require the NetCDF library, used here to read NetCDF files.
+[NetCDF4](https://github.com/Unidata/netcdf4-python) is a Python wrapper for NetCDF, which requires the NetCDF library, used here to read NetCDF files.
  
 Documentation is available [here](http://unidata.github.io/netcdf4-python/).
  
-### pycdf
+- pycdf
  
 [SpacePy](http://pythonhosted.org/SpacePy/index.html) is a python package for space sciences, used here to write CDF files.
  
 Documentation of the package spacepy.pycdf is available [here](http://pythonhosted.org/SpacePy/pycdf.htm).
  
-### Install the Python environment and dependencies
+### Installing the Python environment and dependencies
  
 We will install dependencies in Python environments.
  
-#### If you have NetCDF installed on your machine
+#### Case 1: If you have NetCDF installed on your machine
  
 You can use pip and virtualenv:
  
-    pip install virtualenv
-    virtualenv -p python3 nc2cdf
-    source nc2cdf/bin/activate # Or ". nc2cdf/bin/activate.fish" on Fish terms
-    pip install -r pip_req_nc2cdf.txt
+```bash
+pip install virtualenv
+virtualenv -p python3 nc2cdf
+source nc2cdf/bin/activate # Or ". nc2cdf/bin/activate.fish" on Fish terms
+pip install -r pip_req_nc2cdf.txt
+```
  
-#### If you don't have NetCDF installed on your machine
+#### Case 2: If you don't have NetCDF installed on your machine
  
 The easier way is to use [Anaconda](https://docs.continuum.io/), which is a tool to install compiled Python dependencies in environments.
  
-1. [Install Anaconda3](https://docs.continuum.io/anaconda/install).
-2. Edit your system startup file:
+First, [install Anaconda3](https://docs.continuum.io/anaconda/install).
+
+Then edit your system startup file:
  
 I recommend to add an alias which set the Python path. In this way the Anaconda Python will not be used by default and you can easily deal with multiple Anaconda versions.
  
-At the end of your `~/.bashrc`:
+Add at the end of your `~/.bashrc`:
  
-    alias conda3="set PATH $HOME/.anaconda3/bin/ $PATH"
+```bash
+alias conda3="set PATH $HOME/.anaconda3/bin/ $PATH"
+```
  
 Or on Fish terms (`~/.config/omf/init.fish`):
  
-    alias conda3="set PATH $HOME/.anaconda3/bin/ $PATH; and source $HOME/.anaconda3/etc/fish/conf.d/conda.fish
+```bash
+alias conda3="set PATH $HOME/.anaconda3/bin/ $PATH; and source $HOME/.anaconda3/etc/fish/conf.d/conda.fish
+```
  
-3. Create the environment
+Now create the environment:
  
+```bash
 conda3
 conda create -f conda_env_nc2cdf.yml
+```
  
-4. Activate the Conda environment:
+To finish, activate the Conda environment:
  
+```bash
 source activate nc2cdf # or only "activate nc2cdf" on Fish terms
+```
+
+You can now use the converter.
  
 ### Licence