Wiki
Clone wikikobo-install / Home
Docker Installation Scripts and Instructions
Prerequistes
-
Clone the kobo install repository in
~/src
:git clone https://emaV@bitbucket.org/unhcr/kobo-install.git
-
Clone the kobo dockers repository in~/src
:git clone https://emaV@bitbucket.org/teodorescuserban/kobo-dockers.git
Installation
Edit settings to suit your needs on set_vars
and common.tpl
set_vars
settings
-
set
KOBO_DOMAIN
:unhcr.dev
(in staging),unchr.org
(in production) -
set
KOBO_SSL_KEY
andKOBO_SSL_CRT
: check Configure Nginx to Use SSL -
set
NGINX_HOST
to0.0.0.0
-
set DEFAULT_FILE_STORAGE:
$e DEFAULT_FILE_STORAGE='django.core.files.storage.FileSystemStorage'
diff --git a/set_vars b/set_vars index 8cafa4f..8092a76 100644 --- a/set_vars +++ b/set_vars @@ -14,9 +14,9 @@ $e KOBO_SERVER_IP=$DOCKER0_IP $e KOBO_WB_SERVER_IP=${KOBO_SERVER_IP} $e KOBO_DB_SERVER_IP=${KOBO_SERVER_IP} $e KOBO_PREFIX= -$e KOBO_DOMAIN=humanitarianresponse.info -$e KOBO_SSL_KEY="" -$e KOBO_SSL_CRT="" +$e KOBO_DOMAIN=unhcr.dev +$e KOBO_SSL_KEY="MIIEvwIBADANBgkqhkiG9w0BAQEFAASCBKkwggSlAgEAAoIBAQDXfhEq6SeJ8e0S 21wl6qcO96ed7/c4LioD7kXgKQHnO6Z3aQgySMgiKPXI1Nhl8Zkie08nsAIFfoPB bL3uPsMmsrjPLFMcz95qbVeriMZoQqaZhjO7U3e/HpBDaqOvOW3wkf +$e KOBO_SSL_CRT="MIIDXTCCAkWgAwIBAgIJAPHrtUu6JN3RMA0GCSqGSIb3DQEBCwUAMEUxCzAJBgNV BAYTAkFVMRMwEQYDVQQIDApTb21lLVN0YXRlMSEwHwYDVQQKDBhJbnRlcm5ldCBX aWRnaXRzIFB0eSBMdGQwHhcNMTUxMTE4MTM0NjI4WhcNMTYxMTE3MT $e TRACKJS_TOKEN='' $e GOOGLE_ANALYTICS_TOKEN='' @@ -58,7 +58,7 @@ $e MONGO_HOST=${KOBO_DB_SERVER_IP} $e MONGO_PORT=27017 # -------------- NginX ----------------- -$e NGINX_HOST=${KOBO_WB_SERVER_IP} +$e NGINX_HOST=0.0.0.0 $e NGINX_HTTP_PORT=80 $e NGINX_HTTPS_PORT=443 @@ -86,6 +86,7 @@ $e KOBOCAT_MEDIA_URL=http://${KOBOCAT_PUBLIC_ADDR}/media/ #$e AWS_S3_SECRET_ACCESS_KEY='not-a-real-key' #$e AWS_S3_STORAGE_BUCKET_NAME='kobostatic' #$e DEFAULT_FILE_STORAGE='storages.backends.s3boto.S3BotoStorage' +$e DEFAULT_FILE_STORAGE='django.core.files.storage.FileSystemStorage' $e ENKETO_URL=${ENKETO_SERVER_ADDR} # this is the protocol enketo will try to use to reach our server!
common.tpl
settings
diff --git a/common.tpl b/common.tpl index 685cbd0..4ab2329 100644 --- a/common.tpl +++ b/common.tpl @@ -102,8 +102,8 @@ web: - ./env_nginx # - ./env_secrets ports: - - "${KOBO_WB_SERVER_IP}:${NGINX_HTTP_PORT}:80" - - "${KOBO_WB_SERVER_IP}:${NGINX_HTTPS_PORT}:443" + - "${NGINX_HOST}:${NGINX_HTTP_PORT}:80" + - "${NGINX_HOST}:${NGINX_HTTPS_PORT}:443" volumes: - "${VOL_WB}/static:/srv/www:ro" # get the logs out of glusterfs!
-
double check the settings
-
run
bash prepare.sh
(inspect, change and save if needed, exit) -
docker-compose up -d
-
grab a coffee it will take some time until all images are pulled and containers are started
-
docker exec into kobocat and do:
docker exec -i -t koboinstall_kobocat_1 bash /srv/src/init.sh
-> ONLY ON NEW INSTALLS!- Temporary solution for postgres tables mess
docker exec -i -t koboinstall_kobocat_1 python /srv/src/kobocat/manage.py migrate --fake reversion
docker exec -i -t koboinstall_kobocat_1 python /srv/src/kobocat/manage.py migrate --fake oauth2_provider
docker exec -i -t koboinstall_kobocat_1 python /srv/src/kobocat/manage.py migrate --fake authtoken
docker exec -i -t koboinstall_kobocat_1 python /srv/src/kobocat/manage.py migrate --fake taggit
- rerun
docker exec -i -t koboinstall_kobocat_1 bash /srv/src/init.sh
docker exec -i -t koboinstall_kobocat_1 bash /srv/src/sync_static.sh
-
docker exec into dkobo and do:
docker exec -i -t koboinstall_dkobo_1 bash /srv/src/init.sh
-> ONLY ON NEW INSTALLS!- Temporary solution for postgres tables mess
docker exec -i -t koboinstall_dkobo_1 python /srv/src/koboform/manage.py migrate --fake hub
- rerun
docker exec -i -t koboinstall_dkobo_1 bash /srv/src/init.sh
docker exec -i -t koboinstall_dkobo_1 bash /srv/src/sync_static.sh
docker exec -i -t koboinstall_dkobo_1 bash /srv/src/create_demo_user.sh
-> ONLY FOR DEV / DEMO INSTALLS.
-
docker exec into mongo and do:
docker exec -i -t koboinstall_mongo_1 bash /srv/add_index.sh
<- this will add a missing index to mongo db that will greatly improve speed
-
Install enketo.conf in nginx
Testing environment
Access django admin
https://kobo.unhcr.dev/admin/
Nginx
Access nginx and restart
docker exec -i -t koboinstall_web_1 bash nginx -s reload
nginx configuration koboinstall_web_1
- enketo_http.conf
- enketo_https.conf
- enketo_include.conf
Check nginx SSL certificates if you change the domain
2) if you pull the latest kobo-install, you will see 2 variables in set_vars: KOBO_SSL_KEY and KOBO_SSL_CRT you need to assign to those two the key and certificate followinng this rule: a. get rid of "---- BEGIN---" and "----- END----- " lines b. whats left, put in a single line separated by spaces. those two variables need to be enclosed by double colon.
Step Two — Configure Nginx to Use SSL
-
Login in the host and create the key and the certificate:
openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout ~/src/ssl/nginx.key -out ~/src/ssl/nginx.crt
-
Set the variable in set_vars after converting
\n
to spaces:cat nginx.* | tr '\n' ' '
<old>
* Modify ssl_certificate
and ssl_certificate_key directives
in kobo.conf
in /etc/nginx
sed -i 's/humanitarianresponse_info.bundle/ssl\/nginx/g' /etc/nginx/kobo.conf sed -i 's/humanitarianresponse_info/ssl\/nginx/g' /etc/nginx/kobo.conf
nginx header
Add header
add_header X-Whom Zonker;
Check nginx upstream settings
Using the docker host interface (172.12.42.1:8000) raise an error in koboinstall_web_1 contaniner
.
Set the upstream IP to the container addresses as suggested in using nginx alongside the docker install:
docker inspect -f '{{ .NetworkSettings.IPAddress }}' koboinstall_dkobo_1 docker inspect -f '{{ .NetworkSettings.IPAddress }}' koboinstall_kobocat_1
upstream kobocat { # server 172.17.42.1:8001; server 172.17.0.??:8000; } ... upstream koboform { server 172.17.0.??:8000; # server 172.17.42.1:8000; }
Kobocat
To restart the service
sv restart wsgi
Kobocat settings in /srv/src/koboform.ini
:
accesslog = '/var/log/wsgi/kc.access.log' errorlog = '/var/log/wsgi/kc.error.log' loglevel = 'debug'
Logging in kobocat
import logging logging.basicConfig(filename='/tmp/kc.logging.log',level=logging.DEBUG) logging.info('*** Log init\n\n') logging.debug('\n------------------ XForm'); logging.debug(vars(XForm)); logging.debug('------------------ /XForm\n');
Logging in kobocat (OLD)
Create the directory (mkdir -p /var/log/uwsgi
) and add this in /srv/src/kobocat.ini
:
#location of log files logto = /var/log/uwsgi/%n.log
Kobo
Fix SSL certificate check in dev
If in development server the SSL certificates are self-signed the dkobo application will raise an exception while publishing a form to a new project. From http response:
{"status_code": 504, "detail": "[Errno 1] _ssl.c:510: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed"}
docker exec -i -t koboinstall_dkobo_1 bash
- Remove the validation check on POST request in
survey_draft_views.py
code
diff --git a/dkobo/koboform/views/survey_draft_views.py b/dkobo/koboform/views/survey_draft_views.py index 7af059e..5adbd1d 100644 --- a/dkobo/koboform/views/survey_draft_views.py +++ b/dkobo/koboform/views/survey_draft_views.py @@ -241,7 +241,7 @@ def publish_survey_draft(request, pk, format=None): payload = {u'text_xls_form': valid_xlsform_csv_repr} try: url = kobocat_integration._kobocat_url('/api/v1/forms', internal=True) - response = requests.post(url, headers=headers, data=payload) + response = requests.post(url, headers=headers, data=payload, verify=False) status_code = response.status_code resp = response.json() except Exception, e:
~Fix OfflineGenerationError~
OfflineGenerationError at / You have offline compression enabled but key "eeda3c82d1044337864a1ddf68860b7b" is missing from offline manifest. You may need to run "python manage.py compress".
-
Login in koboinstall_dkobo_1 container:
docker exec -i -t koboinstall_dkobo_1 bash
-
Run
compress
in koboform src directory and resync with nginxcd /srv/src/koboform python manage.py compress bash /srv/src/sync_static.sh
Enketo
-
login
docker exec -i -t koboinstall_enketo_express_1 bash
-
logs
docker logs -f koboinstall_enketo_express_1
-
restart:
kill -2 `ps -elf | grep "\/usr\/bin\/nodejs \/srv\/enketo-express" | cut -b 10-20`
Fix SSL certificate validation
Add to setup/docker/envfile.txt
# Avoid validation of SSL certificate # http://stackoverflow.com/questions/23601989/client-certificate-validation-on-server-side-depth-zero-self-signed-cert-error NODE_TLS_REJECT_UNAUTHORIZED=0;
Add in app.js
:
process.env.NODE_TLS_REJECT_UNAUTHORIZED = "0";
diff --git a/app/lib/communicator/communicator.js b/app/lib/communicator/communicator.js index fc7ebe1..d291784 100644 --- a/app/lib/communicator/communicator.js +++ b/app/lib/communicator/communicator.js @@ -201,6 +201,8 @@ function _request( options ) { var error; var r; var method; +options.rejectUnhauthorized = false; +options.strictSSL = false;
./app/controllers/submission-controller.js:83:options.rejectUnhauthorized = false; ./app/lib/communicator/communicator.js:144:options.rejectUnhauthorized = false; ./app/lib/communicator/communicator.js:217:options.rejectUnhauthorized = false;
- Enketo docker build: emav/enketo_express
Install nginx in the host as proxy
Docker Explained: How To Containerize and Use Nginx as a Proxy
Install nginx
sudo apt-get install -y nginx
sudo mv /etc/nginx/nginx.conf /etc/nginx/nginx.conf.old
Configure nginx: sudo vi /etc//etc/nginx/nginx.conf
daemon off; worker_processes 1; events { worker_connections 1024; } http { sendfile on; gzip on; gzip_http_version 1.0; gzip_proxied any; gzip_min_length 500; gzip_disable "MSIE [1-6]\."; gzip_types text/plain text/xml text/css text/comma-separated-values text/javascript application/x-javascript application/atom+xml; # List of application servers upstream app_http_servers { server 172.17.42.1:80; } upstream app_https_servers { server 172.17.42.1:443; } # Configuration for the server server { # Running port listen 10.9.153.142:80; # Proxying the connections connections location / { proxy_pass http://app_http_servers; proxy_redirect off; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Host $server_name; } } # Configuration for the server server { # Running port listen 10.9.153.142:443 ssl; ssl_certificate /home/emav/src/ssl/nginx.crt; ssl_certificate_key /home/emav/src/ssl/nginx.key; ssl on; ssl_session_cache builtin:1000 shared:SSL:10m; ssl_protocols TLSv1 TLSv1.1 TLSv1.2; ssl_ciphers HIGH:!aNULL:!eNULL:!EXPORT:!CAMELLIA:!DES:!MD5:!PSK:!RC4; ssl_prefer_server_ciphers on; # Proxying the connections connections location / { proxy_pass https://app_https_servers; proxy_redirect off; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Host $server_name; } } }
Enable forwarding with UFW: sudo vi /etc/default/ufw
Scroll down and find the line beginning with DEFAULTFORWARDPOLICY.
Replace: DEFAULT_FORWARD_POLICY="DROP"
With: DEFAULT_FORWARD_POLICY="ACCEPT"
Save and reload the UFW: sudo ufw reload
docker-machine
-
Install docker-machine:
sudo curl -L https://github.com/docker/machine/releases/download/v0.4.0/docker-machine_linux-amd64 > /usr/local/bin/docker-machine sudo chmod +x /usr/local/bin/docker-machine
-
Create docker machine:
docker-machine create --driver=virtualbox kobo-docker-machine
-
Get the IP of the host container:
docker-machine ip kobo-docker-machine
-
Get the right connection information:
eval "$(docker-machine env kobo-docker-machine)"
-
Edit the file, run the composer and needed commands
Docker Machine, Compose, and Swarm: How They Work Together
/misc
-
bin/dockssh
#!/bin/bash docker exec -i -t $1 bash
-
autocomplete dockssh
# set autocomplete for docker ssh complete -W "$(docker ps --format="{{.Names}}" | tr '\n' ' ')" dockssh
Diagram of Docker Running Containers https://docs.google.com/drawings/d/1HQC09jIVo97so-zE22yEvBKBOYfhlwR7yeCJAAa0Jzs/edit
Diagram of Docker Images (including base layers) https://docs.google.com/drawings/d/1LnaWyeC843ddFG9F0JjmjQ_BFtO8qMg2VdGYtpuOKh8/edit
Docker Image-Building Files https://bitbucket.org/teodorescuserban/kobo-dockers
Docker Hub Images
lenovo-emav certificates
$e KOBO_SSL_KEY="MIIEvwIBADANBgkqhkiG9w0BAQEFAASCBKkwggSlAgEAAoIBAQDXfhEq6SeJ8e0S 21wl6qcO96ed7/c4LioD7kXgKQHnO6Z3aQgySMgiKPXI1Nhl8Zkie08nsAIFfoPB bL3uPsMmsrjPLFMcz95qbVeriMZoQqaZhjO7U3e/HpBDaqOvOW3wkfVUHOjIc7pM vgcciVHxW6fPT3APVXpE5S3BdsJvpbiCIETbVi0czFoDB12QykKAFZebRtmFUuBf qpm7cuFNkNe9bKLiM8LgTJOIb++U60MD8UsheO0FsNMmZDLNNo1+xExY9G6BojjK P+gckT07jk+tFwWASasdgAs6L04XLBJQHBJ2wGknXLiQIX2zhQFC8JtCYJzfbBQC 1bnCSsB7AgMBAAECggEAfSzA4jE/g3ThpqN/upK86gfX+PlsnNjMFS5yroyDi4Nt 5k0/sp3ui1FJvJhO/6DAuHPyffZSpkhgVppJFbyldbQ5gDaaLNFevGm5lG3F+shE qEd5g2Yv1/f0CwVGAcIVZ7oRTQPiAJag8HUuAliOHeqE5vqY+H9Jn/2Uf53xwhHq ++HVl+AQyeXq0tvgEvm4N2q4DjmrtHvnYAiz+5pPcwxW2SEZb+n/+qCf7131J3IR obW3jbHmX5xly+tyK0pfeJSGtR/7Jd2TjyLXYpyXkwPNTTJsIT7dLasOzfJhh8X2 UZVL9MJX5LTLr7gUhEvxaKKc1mZ54xYmqAmpysMJQQKBgQD+5ZbNeimpwgt28NVm 1Xco2mHt/JkroWO7Y2MaLGXGEnMrfORxF2FP5BJsL1jbbEeHyHgvswxC6RqYnsWh TaV9l+gOt8I8rzDcmyvA9OWGIknf8I26RAEUqucC2Gjfme74b8O6Jo3IdobIao0+ 3zcLGuGBZzP97gQ29yFdc/WOoQKBgQDYbNH5jpS08TLrHzkJ3vyZltWGTyXnFsuQ vcsiAPAFoQW73sh6aVbAEn0FCELD4lUBbqbi/R48ErW7FRjossjWewEEDIsgQ9sH MXaZGc4CdSL/3ZHOGVsZBeELLDEnrz1elAejmFuW+uVqYIB66wfb94U8dEx8qewx RvW36axFmwKBgQC/k/2w+DViMsb9IWuzO5Ow79oUzfqtkUJtL/jnTiE8KtfHArqL yV/JdQjnqPpnCM4GXwT+DNuVWrAzr3moj7xnVgoDLdfeVmoqFLJg92+fThwqJRnQ upX5W63/NHo2PPMgZJg+TAXaekM5VOROMeSkAxMkeFXab89ZJIs+Ow5/oQKBgQCO jves29dmbK6uo61trB1Vij5yLvZ3htNmLy574oQ4IP3m53PE9lg3QrZpNyzLXO3D eMCjKjndNjHQ3q++r+2NIg7kPOLzyN6gmVtWs+G4SrJYOfnN+XPBQtcfBegHDO5u uzFg+H30MKYncROQY/qUUTHJRY4QW9mCCAzZLSnV2wKBgQCEsSnT2T80j8Yd7TD8 PQK/fHvfaqAb99H3XL1MLXo9lLHuxCP4ClY+96GlclWhn2gZT/hj/wdJBUZEzAav qN8CQApkNlEdkgjavlFoamv1idCqQ37WM1QuzBx2marrSC4gjvhz46k/c7ZOv3Fs GTZZzAQd/I9XVrUUTDQwLBIIUg==" $e KOBO_SSL_CRT="MIIDXTCCAkWgAwIBAgIJAPHrtUu6JN3RMA0GCSqGSIb3DQEBCwUAMEUxCzAJBgNV BAYTAkFVMRMwEQYDVQQIDApTb21lLVN0YXRlMSEwHwYDVQQKDBhJbnRlcm5ldCBX aWRnaXRzIFB0eSBMdGQwHhcNMTUxMTE4MTM0NjI4WhcNMTYxMTE3MTM0NjI4WjBF MQswCQYDVQQGEwJBVTETMBEGA1UECAwKU29tZS1TdGF0ZTEhMB8GA1UECgwYSW50 ZXJuZXQgV2lkZ2l0cyBQdHkgTHRkMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIB CgKCAQEA134RKuknifHtEttcJeqnDvenne/3OC4qA+5F4CkB5zumd2kIMkjIIij1 yNTYZfGZIntPJ7ACBX6DwWy97j7DJrK4zyxTHM/eam1Xq4jGaEKmmYYzu1N3vx6Q Q2qjrzlt8JH1VBzoyHO6TL4HHIlR8Vunz09wD1V6ROUtwXbCb6W4giBE21YtHMxa AwddkMpCgBWXm0bZhVLgX6qZu3LhTZDXvWyi4jPC4EyTiG/vlOtDA/FLIXjtBbDT JmQyzTaNfsRMWPRugaI4yj/oHJE9O45PrRcFgEmrHYALOi9OFywSUBwSdsBpJ1y4 kCF9s4UBQvCbQmCc32wUAtW5wkrAewIDAQABo1AwTjAdBgNVHQ4EFgQUQw9APkhj Xus7zQY9nszf7bqSrrQwHwYDVR0jBBgwFoAUQw9APkhjXus7zQY9nszf7bqSrrQw DAYDVR0TBAUwAwEB/zANBgkqhkiG9w0BAQsFAAOCAQEAsjOiGo7yg9sg6JjLW1zG z8Vvyf3BbJ4R4emxXxQxo56+Dc0A1vqF0FF4TU0UaVhXhZZHc7zZZdXkSAaLLo2/ mNJMvEnNZqG/NHqIkwaeIhLVMoID2AvMAhglpIk6RUAlf4bwEoZg/0e2IuGZnxUd z2fcDjBvDihfbtB3P8LQzrj+3ZuJ1fVhpwfpaVGeeCxeNtYUvXXyFAu9S8QgEhid dM62IzWl29nsKrKOujUChPoRXJubZ8vUsm7i6gdTipO52jU37sb+Qdz0nKSA83Fj V4asnbeKP3dWQPe2o+5663h1auQp6mcCSsc676L7tz/hwsjPo8KMvbCoOXj9T2F7 xg=="
Welcome
Welcome to your wiki! This is the default page we've installed for your convenience. Go ahead and edit it.
Wiki features
This wiki uses the Markdown syntax.
The wiki itself is actually a git repository, which means you can clone it, edit it locally/offline, add images or any other file type, and push it back to us. It will be live immediately.
Go ahead and try:
$ git clone https://unhcr@bitbucket.org/unhcr/kobo-install.git/wiki
Wiki pages are normal files, with the .md extension. You can edit them locally, as well as creating new ones.
Syntax highlighting
You can also highlight snippets of text (we use the excellent Pygments library).
Here's an example of some Python code:
#!python def wiki_rocks(text): formatter = lambda t: "funky"+t return formatter(text)
You can check out the source of this page to see how that's done, and make sure to bookmark the vast library of Pygment lexers, we accept the 'short name' or the 'mimetype' of anything in there.
Have fun!
Updated