CDPP / HELIOPROPA

Commit 5fb9253466fb3bdc88ae386408c33904358810bc

Authored by hitier
1 parent cd18eb81
Exists in rhitier-dev

More documentation

Showing 3 changed files with 117 additions and 39 deletions   Show diff stats
Inline Side-by-side
@@ -22,3 +22,5 @@ venv/* @@ -22,3 +22,5 @@ venv/*
22 VISITS 22 VISITS
23 .ipynb_checkpoints/ 23 .ipynb_checkpoints/
24 config.yml 24 config.yml
  25 +compose.override.yaml
  26 +compose.env
1 # Heliopropa Web 1 # Heliopropa Web
2 2
3 -A web site writen in flask for the visualisation of solarwind propagation simulations. 3 +A website writen in flask for the visualisation of solarwind propagation simulations.
4 4
5 Builds and serves heliopropa's visualizations. 5 Builds and serves heliopropa's visualizations.
6 6
7 It's a `flask` webserver, serving `d3.js` plots. 7 It's a `flask` webserver, serving `d3.js` plots.
  8 +
8 It also gathers NetCDF data from AMDA, and serves it as CSV to the plotter. 9 It also gathers NetCDF data from AMDA, and serves it as CSV to the plotter.
9 10
10 -## Files Overview 11 +## Quick Start
11 12
12 -- `config.yml` : the main configuration file.  
13 -- `web/run.py` : the front controller, holding most of the code.  
14 -- `web/run.log` : you'll find more information about failures in the log.  
15 -- `web/view/home.html.jinja2` : the HTML template.  
16 -- `web/static/js/swapp.ls` : most of the javascript client-side. 13 +You can very quickly have a running app out of the box.
17 14
18 -## Quick Start 15 +Just:
19 16
20 docker compose build 17 docker compose build
21 - docker compose up 18 + docker compose up -d
  19 +
  20 +and point your browser to `http://127.0.0.1:8080/`
  21 +
  22 +See later about configuring port number
22 23
23 -## Docker compose configuration 24 +## Docker
  25 +
  26 +Preferred way of running heliopropa app.
  27 +
  28 +### Configuration
24 29
25 docker stack mainly consists in two files: 30 docker stack mainly consists in two files:
26 31
27 -- `Dockerfile` for the container building  
28 -- `compose.yaml` for the volumes configuration and links with nginx service and gunicorn 32 +- `Dockerfile` : for the container building
  33 +- `compose.yaml` : for the volumes configuration and links with nginx service and gunicorn
29 34
30 -### Volumes configuration and `compose.override.yaml` 35 +
  36 +And your can tweak port configuration thanks to `$PORT` environment variable.
  37 +(@see ports: section in `compose.yaml` )
  38 +
  39 +Either by setting it in your shell
  40 +
  41 + PORT=8081 docker compose up -d
  42 +
  43 +Or in a config file:
  44 +
  45 + cp compose.env-dist compose.env
  46 + ${EDITOR} compose.env # set to your needs
  47 + docker compose --env-file=compose.env up -d
  48 +
  49 +Or
  50 +
  51 + cp compose.env-dist .env
  52 + ${EDITOR} .env # set to your needs
  53 + docker compose up
  54 +
  55 +### Volumes and `compose.override.yaml`
31 56
32 You may want to understand where data is stored: 57 You may want to understand where data is stored:
33 58
34 -The web application saves `*.csv` files in the `./cache/` directory, which happens to be a docker volume named `helio_cache` 59 +The web application saves `*.csv` files in the `./cache/` directory, which happens to be a docker volume
  60 +named `helio_cache`.
35 61
36 -The speasy library holds a cache dir located at `container:/root/.cache/speasy/Cache/` which is also mounted as a volume: `speasy_cache` 62 +The speasy library holds a cache dir located at `container:/root/.cache/speasy/Cache/` which is also mounted as a
  63 +volume: `speasy_cache`.
37 64
38 Two configuration files are also used: 65 Two configuration files are also used:
39 66
40 -- `./config.yml` for the web app  
41 -- `${HOME}/.config/speasy/config.ini` for the speasy library 67 +- `./config.yml` : for the web app.
  68 +- `${HOME}/.config/speasy/config.ini` : for the speasy library.
42 69
  70 +You can use the `compose.override.yaml` file to make dynamic links between your local files or directories and the
  71 +container.
43 72
44 -You can use the `compose.override.yaml` file to make dynamic links between your local files and the container.  
45 That way you can live edit the config files, and also use already loaded cache directories. 73 That way you can live edit the config files, and also use already loaded cache directories.
46 74
  75 +To do so:
  76 +
47 cp compose.override.yaml-dist compose.override.yaml 77 cp compose.override.yaml-dist compose.override.yaml
48 ${EDITOR} compose.override.yaml # make sure its ok, comment what you dont need 78 ${EDITOR} compose.override.yaml # make sure its ok, comment what you dont need
49 docker compose up -d 79 docker compose up -d
50 80
51 -## Develop 81 +## Developer
  82 +
  83 +### Files Overview
  84 +
  85 +- `config.yml` : the main configuration file.
  86 +- `web/run.py` : the front controller, holding most of the code.
  87 +- `web/run.log` : you'll find more information about failures in the log.
  88 +- `web/view/home.html.jinja2` : the HTML template.
  89 +- `web/static/js/main.js` : most of the javascript client-side.
52 90
53 ### Configuration 91 ### Configuration
54 92
55 -``` bash  
56 -cp config.yml-dist config.yml  
57 -$(EDITOR) config.yml # but defaults should be ok  
58 -``` 93 + cp config.yml-dist config.yml
  94 + $(EDITOR) config.yml # but defaults should be ok
  95 +
  96 +Also take a look at the `./ressources/` directory for more available configuration files.
59 97
60 ### Python venv 98 ### Python venv
61 99
62 -``` bash  
63 -python3 -m venv venv  
64 -source venv/bin/activate  
65 -pip install -r requirements.txt  
66 -``` 100 + python3 -m venv venv
  101 + source venv/bin/activate
  102 + pip install -r requirements.txt
67 103
68 ### Run locally 104 ### Run locally
69 -``` bash  
70 -source venv/bin/activate  
71 -DEBUG=true python web/run.py  
72 -``` 105 +
  106 + source venv/bin/activate
  107 + DEBUG=true python web/run.py
73 108
74 Then, browse [localhost:5000](http://localhost:5000). 109 Then, browse [localhost:5000](http://localhost:5000).
75 110
76 -## Gunicorn and Nginx deployment on OVH for development 111 +### Admin task
77 112
78 -see [ Digital Ocean documentation ] (https://www.digitalocean.com/community/tutorials/how-to-serve-flask-applications-with-gunicorn-and-nginx-on-ubuntu-20-04-fr) 113 +A tool called `hp_cmd.py` is available to make so administrative task.
  114 +
  115 +It allows you to nightly populate the csv cache so later visitors won't have to wait for data download.
  116 +
  117 +Basic usage:
  118 +
  119 + # look at what it would do
  120 + python web/hp_cmd.py --dry-run -clog csv_gen -a
  121 +
  122 + # do it
  123 + python web/hp_cmd.py -clog csv_gen -a
  124 +
  125 + # see all usages
  126 + python web/hp_cmd.py --help
  127 +
  128 +
  129 +Docker usage:
  130 +
  131 + docker compose exec web python web/hp_cmd.py --all
  132 +
  133 +
  134 +Http tasks:
79 135
  136 + http://${HOSTNAME}/cache/clear
  137 + http://${HOSTNAME}/cache/cleanup
  138 + http://${HOSTNAME}/cache/warmup
  139 + http://${HOSTNAME}/log
  140 + http://${HOSTNAME}/log/clear
  141 +
  142 +### Notebooks
  143 +
  144 +If your want to dig into code usage take a look at our notebooks:
  145 +
  146 + pip install -r requirements-nb.txt
  147 + jupyter notebook ./notebooks/
  148 +
  149 +then browse to given url.
  150 +
  151 +Each file name is self-explanatory, and each notebook contains explanations.
  152 +
  153 +
  154 +## Gunicorn and Nginx deployment
  155 +
  156 +see [ Digital Ocean documentation ] (https://www.digitalocean.com/community/tutorials/how-to-serve-flask-applications-with-gunicorn-and-nginx-on-ubuntu-20-04-fr)
80 157
81 ### wep app service and Gunicorn 158 ### wep app service and Gunicorn
82 159
@@ -114,19 +191,17 @@ see ./resources/heliopropa.nginx @@ -114,19 +191,17 @@ see ./resources/heliopropa.nginx
114 } 191 }
115 } 192 }
116 193
117 -  
118 ### Start and Allow at system reboot 194 ### Start and Allow at system reboot
119 195
120 sudo systemctl start nginx 196 sudo systemctl start nginx
121 sudo systemctl start heliopropa 197 sudo systemctl start heliopropa
122 sudo systemctl enable heliopropa 198 sudo systemctl enable heliopropa
123 199
124 -  
125 ## Git hook auto deploy 200 ## Git hook auto deploy
126 201
127 see ./resources/post-receive.git-hook 202 see ./resources/post-receive.git-hook
128 203
129 -Set bare repo with hook on vps 204 +Set bare repo with hook on vps
130 205
131 ssh user@vps 206 ssh user@vps
132 git clone --bare path-to-heliopropa-repo heliopropa.git 207 git clone --bare path-to-heliopropa-repo heliopropa.git
@@ -134,8 +209,6 @@ Set bare repo with hook on vps @@ -134,8 +209,6 @@ Set bare repo with hook on vps
134 vim post-receive 209 vim post-receive
135 chmod +x post-receive 210 chmod +x post-receive
136 211
137 -  
138 -  
139 Push from dev computer 212 Push from dev computer
140 213
141 git remote add helio-dev user@vps:heliopropa.git 214 git remote add helio-dev user@vps:heliopropa.git
config.yml-dist
  Diff comments   View file @5fb9253
@@ -912,6 +912,7 @@ targets: @@ -912,6 +912,7 @@ targets:
912 atse: 'psp_sta_da' 912 atse: 'psp_sta_da'
913 locked: false 913 locked: false
914 default: false 914 default: false
  915 +
915 # - type: 'probe' 916 # - type: 'probe'
916 # slug: 'rosetta' 917 # slug: 'rosetta'
917 # name: 'Rosetta' 918 # name: 'Rosetta'
@@ -935,6 +936,7 @@ targets: @@ -935,6 +936,7 @@ targets:
935 # - slug: 'tao_ros_sw' 936 # - slug: 'tao_ros_sw'
936 # locked: true 937 # locked: true
937 # default: false 938 # default: false
  939 +
938 # - type: 'probe' 940 # - type: 'probe'
939 # slug: 'juno' 941 # slug: 'juno'
940 # name: 'Juno' 942 # name: 'Juno'
@@ -954,6 +956,7 @@ targets: @@ -954,6 +956,7 @@ targets:
954 # - slug: 'tao_juno_sw' 956 # - slug: 'tao_juno_sw'
955 # locked: true 957 # locked: true
956 # default: false 958 # default: false
  959 +
957 # - type: 'comet' 960 # - type: 'comet'
958 # slug: 'p67' 961 # slug: 'p67'
959 # name: 'Churyumov-Gerasimenko' 962 # name: 'Churyumov-Gerasimenko'