Commit ef9a876714bffb1a2e8415c91020c9d4d8d3aac7

Authored by Nathanael Jourdane
1 parent 5ab287f2
Exists in master

Add the CDF specs

Showing 2 changed files with 471 additions and 30 deletions   Show diff stats
CDF_spec.md 0 → 100644
@@ -0,0 +1,420 @@ @@ -0,0 +1,420 @@
  1 +# Formating CDF files for AMDADB
  2 +
  3 +This specification follows the ISTP guidelines, see [the ISTP guide](https://spdf.gsfc.nasa.gov/istp_guide/).
  4 +
  5 +---
  6 +
  7 +## Global CDF attributes
  8 +
  9 +See [the ISTP guidelines for global attributes](https://spdf.gsfc.nasa.gov/istp_guide/gattributes.html).
  10 +
  11 +### Acknowledgement
  12 +
  13 +Value of node `NumericalData/ResourceHeader/Acknowledgement` from SPASE file *NumericalData* corresponding to the granule.
  14 +
  15 +> **Ex:**
  16 +>
  17 +> NASA, Prof. Henri Reme, European Space Agency
  18 +
  19 +### Data_type
  20 +
  21 +Dataset name
  22 +
  23 +> **Ex:**
  24 +>
  25 +> PRP
  26 +
  27 +### Data_version
  28 +
  29 + 1
  30 +
  31 +### Descriptor
  32 +
  33 +Instrument name
  34 +
  35 +> **Ex:**
  36 +>
  37 +> CIS
  38 +
  39 +### Discipline
  40 +
  41 +- If target name is `sun` or `earth` : ***Space Physics***
  42 +- otherwise, look for node value of `NumericalData/ObservedRegion` from the *NumericalData* SPASE file corresponding to the granule:
  43 + - if it's `Magnetosphere`: ***Space Physics>Magnetospheric Science***;
  44 + - if it's `NearSurface`: ***Space Physics>Ionospheric Science***;
  45 + - if it's `Heliosphere`: ***Space Physics>Interplanetary Studies***.
  46 +
  47 +### Generated_by
  48 +
  49 + CDPP>Centre de Donnees de la Physique des Plasmas
  50 +
  51 +### Generation_date
  52 +
  53 +Value of node `NumericalData/ResourceHeader/ReleaseDate` from the *NumericalData* SPASE file corresponding to the granule, formated as *yyyymmdd*.
  54 +
  55 +> **Ex:**
  56 +>
  57 +> 20151019
  58 +
  59 +### HTTP_LINK
  60 +
  61 + http://amda.cdpp.eu
  62 +
  63 +### Instrument_type
  64 +
  65 +Value of node `Instrument/InstrumentType` from the *Instrument* SPASE file corresponding to the granule.
  66 +
  67 +> **Ex:**
  68 +>
  69 +> ElectrostaticAnalyser
  70 +
  71 +### LINK_TEXT
  72 +
  73 +Value of node `NumericalData/ResourceHeader/AlternateName` from the *NumericalData* SPASE file corresponding to the granule **+** ` data are available at `.
  74 +
  75 +> **Ex:**
  76 +>
  77 +> Cluster 1 Prime Parameter CIS data data are available at
  78 +
  79 +### LINK_TITLE
  80 +
  81 + CDPP AMDA DATABASE
  82 +
  83 +### Logical_file_id
  84 +
  85 +Dataset name **+** `_` **+** `start_time` epn-core value, formated as `YYYYMMDDHHMMSS`.
  86 +
  87 +Note: This is actually the file name without extension.
  88 +
  89 +> **Ex**
  90 +>
  91 +> CLUST1_CIS_PRP_19900302040000
  92 +
  93 +### Logical_source
  94 +
  95 +Dataset name
  96 +
  97 +> **Ex:**
  98 +>
  99 +> CLUST1_CIS_PRP
  100 +
  101 +### Logical_source_description
  102 +
  103 +Value of node `NumericalData/ResourceHeader/Description` from the *NumericalData* SPASE file corresponding to the granule.
  104 +
  105 +> **Ex:**
  106 +>
  107 +> This instrument (CIS: Cluster Ion Spectrometry) is capable of obtaining full 3D ion distributions with high time resolution (in one spacecraft spin) and mass-per-charge resolution. The experiment consists of [...]
  108 +
  109 +### Mission_group
  110 +
  111 +- Last element of the value of node `Observatory/ObservatoryGroupID` from the *Observatory* SPASE file corresponding to the granule.
  112 +- if the node doesn't exist: value of node `Observatory/ResourceName`.
  113 +
  114 +> **Ex:**
  115 +>
  116 +> Cluster
  117 +
  118 +### MODS
  119 +
  120 + Extracted from NetCDF AMDA internal data.
  121 +
  122 +### Parents
  123 +
  124 +The NetCDF file name.
  125 +
  126 +> **Ex**
  127 +>
  128 +> CLUST1_CIS_PRP_19900302040000.nc
  129 +
  130 +### PI_affiliation
  131 +
  132 +Value of node `Person/OrganisationName` from the *Person* SPASE file corresponding to the granule.
  133 +
  134 +> **Ex**
  135 +>
  136 +> ESTEC, ESA
  137 +
  138 +### PI_name
  139 +
  140 +Value of node `NumericalData/Contact/PersonID` where `Role` node value is `PrincipalInvestigator`, from the *NumericalData* SPASE file corresponding to the granule.
  141 +
  142 +> **Ex:**
  143 +>
  144 +> Henri.Reme
  145 +
  146 +### Project
  147 +
  148 +A list with:
  149 +
  150 +- `CDPP>Centre de Donnees de la Physique des Plasmas`
  151 +- the value of node `NumericalData/ProviderName`, from the *NumericalData* SPASE file corresponding to the granule.
  152 +
  153 +> **Ex:**
  154 +>
  155 +> - CDPP>Centre de Donnees de la Physique des Plasmas
  156 +> - CSDS-CNES
  157 +
  158 +### Rules_of_use
  159 +
  160 + System Use Policy
  161 +
  162 + Thank you for acknowledging the use of AMDA in publications with wording like "Data analysis was performed with the AMDA science analysis system provided by the Centre de Données de la Physique des Plasmas (CDPP) supported by CNRS, CNES, Observatoire de Paris and Université Paul Sabatier, Toulouse."
  163 +
  164 + Data use policy from originating <NAME> data center: <URL>
  165 +
  166 +where :
  167 +- <NAME> is the value of node `NumericalData/ProviderName` from the SPASE file *NumericalData* corresponding to the granule, or CDPP if the value doesn't exist.
  168 +- and <URL> is kept from this dictionary (according to <NAME>).
  169 +
  170 +- CDPP: http://amda.irap.omp.eu/help/licenceAction.do.html
  171 +- CDAWEB: http://spdf.gsfc.nasa.gov/new_users.html
  172 +- PDS: http://ppi.pds.nasa.gov/citations_policy.jsp
  173 +- MAPSKP: http://mapskp.cesr.fr/use_policy
  174 +- THEMIS: http://themis.ssl.berkeley.edu/roadrules.shtml
  175 +- MAVEN: http://lasp.colorado.edu/home/maven/files/2014/09/MAVEN_public-rules-of-the-road-v3.pdf
  176 +- CASSINI/RPWS/SKR: http://amda.irap.omp.eu/help/SKR.html
  177 +- VEXGRAZ: http://amda.irap.omp.eu/help/VEXGRAZ.html
  178 +- CAA: http://caa.estec.esa.int/caa/ror.xml
  179 +- CSA: http://www.cosmos.esa.int/web/csa/rules-of-the-road
  180 +- STEREO: http://amda.irap.omp.eu/help/STEREO.html
  181 +- CCMC: http://ccmc.gsfc.nasa.gov/PubPolicy.php
  182 +
  183 +
  184 +> **Ex:**
  185 +>
  186 +> System Use Policy
  187 +>
  188 +> Thank you for acknowledging the use of AMDA in publications with wording like "Data analysis was performed with the AMDA science analysis system provided by the Centre de Données de la Physique des Plasmas (CDPP) supported by CNRS, CNES, Observatoire de Paris and Université Paul Sabatier, Toulouse."
  189 +>
  190 +> Data use policy from originating CDPP data center: http://amda.irap.omp.eu/help/licenceAction.do.html
  191 +
  192 +### Software_version
  193 +
  194 +Version of the NetCDF to CDF converter.
  195 +
  196 +### Source_name
  197 +
  198 +The mission name
  199 +
  200 +> **Ex:**
  201 +>
  202 +> clust1
  203 +
  204 +### TEXT
  205 +
  206 +The value of node `NumericalData/ResourceHeader/InformationURL/Name` **+** `, ` **+** the value of node `NumericalData/ResourceHeader/InformationURL/URL`, from the SPASE file *NumericalData* corresponding to the granule.
  207 +
  208 +> **Ex:**
  209 +>
  210 +> NSSDC Master Catalog listing for Cluster II Rumba Cluster Ion Spectrometry (CIS), http://nssdc.gsfc.nasa.gov/nmc/experimentDisplay.do?id=2000-045A-02
  211 +
  212 +### Time_resolution
  213 +
  214 +`time_sampling_step` value of epn-core, in seconds + ` seconds`.
  215 +
  216 +> **Ex:**
  217 +>
  218 +> 3 seconds
  219 +
  220 +### TITLE
  221 +
  222 +The value of node `NumericalData/ResourceHeader/AlternateName`, from the SPASE file *NumericalData* corresponding to the granule.
  223 +
  224 +> **Ex:**
  225 +>
  226 +> Cluster 1 Prime Parameter CIS data
  227 +
  228 +### PDS_Start_time
  229 +
  230 +`time_min` value of epn-core, in ISO time.
  231 +
  232 +> **Ex:**
  233 +>
  234 +> 2007-02-13T09:08:43Z
  235 +
  236 +### PDS_Stop_time
  237 +
  238 +`time_max` value of epn-core, in ISO time.
  239 +
  240 +> **Ex:**
  241 +>
  242 +> 2007-02-14T09:08:43Z
  243 +
  244 +### PDS_Observation_target
  245 +
  246 +`target_name` value of epn-core. Use a list of values if necessary (split the `#` character).
  247 +
  248 +> **Ex:**
  249 +>
  250 +> Saturn
  251 +
  252 +### Observation_type
  253 +
  254 +Look for the child node in `NumericalData/Parameter` node, from the SPASE file *NumericalData* corresponding to the granule.
  255 +
  256 +It is one of: `Field`, `Particules`, `Wave`, `Mixed`, `Support`.
  257 +
  258 +> **Ex:**
  259 +>
  260 +> Particle
  261 +
  262 +### Other epn-core parameters
  263 +
  264 +Add other epn-core parameters, with `VESPA_` prefix, only if:
  265 +- the value is not empty;
  266 +- this is not the same value as the other element of a tuple (ie min/max).
  267 +
  268 +> **Ex:**
  269 +>
  270 +> - VESPA_spase_measurement_type: EnergeticParticles
  271 +> - VESPA_spatial_frame_type: cartesian
  272 +
  273 +---
  274 +
  275 +## Variable attributes (TBC)
  276 +
  277 +See [the ISTP guidelines for variables](https://spdf.gsfc.nasa.gov/istp_guide/variables.html).
  278 +
  279 +To create new variables with [spacepy.pycdf)(http://pythonhosted.org/SpacePy/pycdf.html), use [cdf.new()](http://pythonhosted.org/SpacePy/autosummary/spacepy.pycdf.CDF.html#spacepy.pycdf.CDF.new) and not `attr[]`.
  280 +
  281 +To name variables, use the `NumericalData/Parameter/ParameterKey` parameter from the SPASE file.
  282 +
  283 +### `CATDESC`
  284 +
  285 +type: CDF_CHAR
  286 +
  287 +The value of node `NumericalData/Parameter/Description`.
  288 +
  289 +> **Ex**
  290 +>
  291 +> Flux density spectrogram measured on the RH polarized array.
  292 +
  293 +### DEPEND_0
  294 +
  295 +type : CDF_CHAR
  296 +
  297 + Epoch
  298 +
  299 +### LABL_PTR_1
  300 +
  301 +Note: Only if the node value of `NumericalData/Parameter/RenderingHints` is `time series` and the node `NumericalData/Parameter/Structure` exist.
  302 +
  303 +Note 2: `LABL_PTR_1` and `DEPEND_1` can not are both presents.
  304 +
  305 +**Step 1:** Get the labels, which are the nodes value of `NumericalData/Parameter/Structure/Element/Name`.
  306 +
  307 +> **Ex (for skr151150000.nc.gz):**
  308 +>
  309 +> - RH 100-400kHz
  310 +> - LH 100-400kHz
  311 +> - RH 10-1000kHz
  312 +> - RH 10-1000kHz
  313 +
  314 +**Step 2:** Create a new variable stored with these labels. Then `LABL_PTR_1` is the ID of this variable.
  315 +
  316 +### DEPEND_1
  317 +
  318 +Note: Only if the node value of `NumericalData/Parameter/RenderingHints` is `spectrogram`.
  319 +
  320 +Note 2: `LABL_PTR_1` and `DEPEND_1` can not are both presents.
  321 +
  322 +type : CDF_CHAR
  323 +
  324 +From the file `<dataset_name>_info` from the dataset directory, look for `energy` or `frequency` variables value.
  325 +
  326 +### DICT_KEY
  327 +
  328 +type: CDF_CHAR
  329 +
  330 +> **Ex:**
  331 +>
  332 +> electric_field>power
  333 +
  334 +### DISPLAY_TYPE
  335 +
  336 +type: CDF_CHAR
  337 +
  338 +> **Ex:**
  339 +>
  340 +> time_series
  341 +
  342 +### FIELDNAM
  343 +
  344 +type: CDF_CHAR
  345 +
  346 +> **Ex:**
  347 +>
  348 +> RR
  349 +
  350 +### FILLVAL
  351 +
  352 +type: CDF_UINT1
  353 +
  354 +> **Ex:**
  355 +>
  356 +> 255
  357 +
  358 +### LABLAXIS
  359 +
  360 +type: CDF_CHAR
  361 +
  362 +> **Ex:**
  363 +>
  364 +> RH flux density
  365 +
  366 +### UNITS
  367 +
  368 +type: CDF_CHAR
  369 +
  370 +> **Ex:**
  371 +>
  372 +> dB
  373 +
  374 +### VALIDMIN
  375 +
  376 +type: CDF_UINT1
  377 +
  378 +> **Ex:**
  379 +>
  380 +> 0
  381 +
  382 +### VALIDMAX
  383 +
  384 +type: CDF_UINT1
  385 +
  386 +> **Ex:**
  387 +>
  388 +> 255
  389 +
  390 +### VAR_TYPE
  391 +
  392 +type: CDF_CHAR
  393 +
  394 +> **Ex:**
  395 +>
  396 +> data
  397 +
  398 +### SCALETYP
  399 +
  400 +type: CDF_CHAR
  401 +
  402 +> **Ex:**
  403 +>
  404 +> lin
  405 +
  406 +### SCALEMIN
  407 +
  408 +type: CDF_UINT1
  409 +
  410 +> **Ex:**
  411 +>
  412 +> 0
  413 +
  414 +### SCALEMAX
  415 +
  416 +> **Ex:**
  417 +>
  418 +> 255
  419 +
  420 +### UCD
1 # CDF tools 1 # CDF tools
2 2
3 -## NetCDF to CDF converter 3 +## NetCDF to CDF converter for AMDADB
4 4
5 - File: [nc2cdf.py](./nc2cdf.py) 5 - File: [nc2cdf.py](./nc2cdf.py)
6 - Python interpreter: 3.6 6 - Python interpreter: 3.6
@@ -11,87 +11,108 @@ @@ -11,87 +11,108 @@
11 11
12 Convert the NetCDF file, save it in a temp directory, then display its path: 12 Convert the NetCDF file, save it in a temp directory, then display its path:
13 13
14 - ./nc2cdf.py path/to/input_file.nc.gz 14 +```bash
  15 +./nc2cdf.py path/to/input_file.nc.gz
  16 +```
15 17
16 Convert a Net-CDF file and save it in the specified path. 18 Convert a Net-CDF file and save it in the specified path.
17 19
18 - ./nc2cdf.py path/to/input_file.nc.gz path/to/output_file.cdf 20 +```bash
  21 +./nc2cdf.py path/to/input_file.nc.gz path/to/output_file.cdf
  22 +```
19 23
20 -**Note:** If the specified input file is a gzip archive, it will be automatically extracted before the conversion. 24 +**Note:** If the specified input file is a gzip archive, it will be automatically extracted in a temp directory before the conversion.
21 25
22 #### Describing a NetCDf file: 26 #### Describing a NetCDf file:
23 27
24 - ./nc2cdf.py -i path/to/file.nc.gz 28 +```bash
  29 +./nc2cdf.py -i path/to/file.nc.gz
  30 +```
25 31
26 This display information about a Net-CDF file (such as global attributes and variables information). 32 This display information about a Net-CDF file (such as global attributes and variables information).
27 33
28 -### Python library usage 34 +### Python usage
29 35
30 - import nc2cdf 36 +```python
  37 +import nc2cdf
31 38
32 - netcdf = NetCdf('path/to/input_file.nc.gz') 39 +netcdf = NetCdf('path/to/input_file.nc.gz')
33 40
34 - netcdf.describe() 41 +netcdf.describe()
35 42
36 - netcdf.get_cdf()  
37 - print('CDF path: ' + netcdf.get_cdf_path()) 43 +netcdf.get_cdf()
  44 +print('CDF path: ' + netcdf.get_cdf_path())
38 45
39 - netcdf.get_cdf('path/to/output_file.cdf') 46 +netcdf.get_cdf('path/to/output_file.cdf')
  47 +```
40 48
41 -## About NetCDF4 and pycdf 49 +## Dependencies
42 50
43 -### NetCDF4 51 +- NetCDF4
44 52
45 [NetCDF](https://www.unidata.ucar.edu/software/netcdf/) is C library to read and edit NetCDF files. 53 [NetCDF](https://www.unidata.ucar.edu/software/netcdf/) is C library to read and edit NetCDF files.
46 54
47 -[NetCDF4](https://github.com/Unidata/netcdf4-python) is a Python wrapper for NetCDF, which require the NetCDF library, used here to read NetCDF files. 55 +[NetCDF4](https://github.com/Unidata/netcdf4-python) is a Python wrapper for NetCDF, which requires the NetCDF library, used here to read NetCDF files.
48 56
49 Documentation is available [here](http://unidata.github.io/netcdf4-python/). 57 Documentation is available [here](http://unidata.github.io/netcdf4-python/).
50 58
51 -### pycdf 59 +- pycdf
52 60
53 [SpacePy](http://pythonhosted.org/SpacePy/index.html) is a python package for space sciences, used here to write CDF files. 61 [SpacePy](http://pythonhosted.org/SpacePy/index.html) is a python package for space sciences, used here to write CDF files.
54 62
55 Documentation of the package spacepy.pycdf is available [here](http://pythonhosted.org/SpacePy/pycdf.htm). 63 Documentation of the package spacepy.pycdf is available [here](http://pythonhosted.org/SpacePy/pycdf.htm).
56 64
57 -### Install the Python environment and dependencies 65 +### Installing the Python environment and dependencies
58 66
59 We will install dependencies in Python environments. 67 We will install dependencies in Python environments.
60 68
61 -#### If you have NetCDF installed on your machine 69 +#### Case 1: If you have NetCDF installed on your machine
62 70
63 You can use pip and virtualenv: 71 You can use pip and virtualenv:
64 72
65 - pip install virtualenv  
66 - virtualenv -p python3 nc2cdf  
67 - source nc2cdf/bin/activate # Or ". nc2cdf/bin/activate.fish" on Fish terms  
68 - pip install -r pip_req_nc2cdf.txt 73 +```bash
  74 +pip install virtualenv
  75 +virtualenv -p python3 nc2cdf
  76 +source nc2cdf/bin/activate # Or ". nc2cdf/bin/activate.fish" on Fish terms
  77 +pip install -r pip_req_nc2cdf.txt
  78 +```
69 79
70 -#### If you don't have NetCDF installed on your machine 80 +#### Case 2: If you don't have NetCDF installed on your machine
71 81
72 The easier way is to use [Anaconda](https://docs.continuum.io/), which is a tool to install compiled Python dependencies in environments. 82 The easier way is to use [Anaconda](https://docs.continuum.io/), which is a tool to install compiled Python dependencies in environments.
73 83
74 -1. [Install Anaconda3](https://docs.continuum.io/anaconda/install).  
75 -2. Edit your system startup file: 84 +First, [install Anaconda3](https://docs.continuum.io/anaconda/install).
  85 +
  86 +Then edit your system startup file:
76 87
77 I recommend to add an alias which set the Python path. In this way the Anaconda Python will not be used by default and you can easily deal with multiple Anaconda versions. 88 I recommend to add an alias which set the Python path. In this way the Anaconda Python will not be used by default and you can easily deal with multiple Anaconda versions.
78 89
79 -At the end of your `~/.bashrc`: 90 +Add at the end of your `~/.bashrc`:
80 91
81 - alias conda3="set PATH $HOME/.anaconda3/bin/ $PATH" 92 +```bash
  93 +alias conda3="set PATH $HOME/.anaconda3/bin/ $PATH"
  94 +```
82 95
83 Or on Fish terms (`~/.config/omf/init.fish`): 96 Or on Fish terms (`~/.config/omf/init.fish`):
84 97
85 - alias conda3="set PATH $HOME/.anaconda3/bin/ $PATH; and source $HOME/.anaconda3/etc/fish/conf.d/conda.fish 98 +```bash
  99 +alias conda3="set PATH $HOME/.anaconda3/bin/ $PATH; and source $HOME/.anaconda3/etc/fish/conf.d/conda.fish
  100 +```
86 101
87 -3. Create the environment 102 +Now create the environment:
88 103
  104 +```bash
89 conda3 105 conda3
90 conda create -f conda_env_nc2cdf.yml 106 conda create -f conda_env_nc2cdf.yml
  107 +```
91 108
92 -4. Activate the Conda environment: 109 +To finish, activate the Conda environment:
93 110
  111 +```bash
94 source activate nc2cdf # or only "activate nc2cdf" on Fish terms 112 source activate nc2cdf # or only "activate nc2cdf" on Fish terms
  113 +```
  114 +
  115 +You can now use the converter.
95 116
96 ### Licence 117 ### Licence
97 118