1. UNHCR Avatar UNHCR
  2. KOBO
  3. kobo-install

Wiki

Clone wiki

kobo-install / Home

View History

Docker Installation Scripts and Instructions

Prerequistes

Installation

Edit settings to suit your needs on set_vars and common.tpl

set_vars settings

  • set KOBO_DOMAIN: unhcr.dev (in staging), unchr.org (in production)

  • set KOBO_SSL_KEY and KOBO_SSL_CRT: check Configure Nginx to Use SSL

  • set NGINX_HOST to 0.0.0.0

  • set DEFAULT_FILE_STORAGE: $e DEFAULT_FILE_STORAGE='django.core.files.storage.FileSystemStorage'

diff --git a/set_vars b/set_vars
index 8cafa4f..8092a76 100644
--- a/set_vars
+++ b/set_vars
@@ -14,9 +14,9 @@ $e KOBO_SERVER_IP=$DOCKER0_IP
 $e KOBO_WB_SERVER_IP=${KOBO_SERVER_IP}
 $e KOBO_DB_SERVER_IP=${KOBO_SERVER_IP}
 $e KOBO_PREFIX=
-$e KOBO_DOMAIN=humanitarianresponse.info
-$e KOBO_SSL_KEY=""
-$e KOBO_SSL_CRT=""
+$e KOBO_DOMAIN=unhcr.dev
+$e KOBO_SSL_KEY="MIIEvwIBADANBgkqhkiG9w0BAQEFAASCBKkwggSlAgEAAoIBAQDXfhEq6SeJ8e0S 21wl6qcO96ed7/c4LioD7kXgKQHnO6Z3aQgySMgiKPXI1Nhl8Zkie08nsAIFfoPB bL3uPsMmsrjPLFMcz95qbVeriMZoQqaZhjO7U3e/HpBDaqOvOW3wkf
+$e KOBO_SSL_CRT="MIIDXTCCAkWgAwIBAgIJAPHrtUu6JN3RMA0GCSqGSIb3DQEBCwUAMEUxCzAJBgNV BAYTAkFVMRMwEQYDVQQIDApTb21lLVN0YXRlMSEwHwYDVQQKDBhJbnRlcm5ldCBX aWRnaXRzIFB0eSBMdGQwHhcNMTUxMTE4MTM0NjI4WhcNMTYxMTE3MT
 $e TRACKJS_TOKEN=''
 $e GOOGLE_ANALYTICS_TOKEN=''

@@ -58,7 +58,7 @@ $e MONGO_HOST=${KOBO_DB_SERVER_IP}
 $e MONGO_PORT=27017

 # -------------- NginX -----------------
-$e NGINX_HOST=${KOBO_WB_SERVER_IP}
+$e NGINX_HOST=0.0.0.0
 $e NGINX_HTTP_PORT=80
 $e NGINX_HTTPS_PORT=443

@@ -86,6 +86,7 @@ $e KOBOCAT_MEDIA_URL=http://${KOBOCAT_PUBLIC_ADDR}/media/
 #$e AWS_S3_SECRET_ACCESS_KEY='not-a-real-key'
 #$e AWS_S3_STORAGE_BUCKET_NAME='kobostatic'
 #$e DEFAULT_FILE_STORAGE='storages.backends.s3boto.S3BotoStorage'
+$e DEFAULT_FILE_STORAGE='django.core.files.storage.FileSystemStorage'

 $e ENKETO_URL=${ENKETO_SERVER_ADDR}
 # this is the protocol enketo will try to use to reach our server!

common.tpl settings 

diff --git a/common.tpl b/common.tpl
index 685cbd0..4ab2329 100644
--- a/common.tpl
+++ b/common.tpl
@@ -102,8 +102,8 @@ web:
     - ./env_nginx
   #  - ./env_secrets
   ports:
-    - "${KOBO_WB_SERVER_IP}:${NGINX_HTTP_PORT}:80"
-    - "${KOBO_WB_SERVER_IP}:${NGINX_HTTPS_PORT}:443"
+    - "${NGINX_HOST}:${NGINX_HTTP_PORT}:80"
+    - "${NGINX_HOST}:${NGINX_HTTPS_PORT}:443"
   volumes:
       - "${VOL_WB}/static:/srv/www:ro"
       # get the logs out of glusterfs!

  1. double check the settings

  2. run bash prepare.sh (inspect, change and save if needed, exit)

  3. docker-compose up -d

  4. grab a coffee it will take some time until all images are pulled and containers are started

  5. docker exec into kobocat and do:

    • docker exec -i -t koboinstall_kobocat_1 bash /srv/src/init.sh -> ONLY ON NEW INSTALLS!
    • Temporary solution for postgres tables mess
      • docker exec -i -t koboinstall_kobocat_1 python /srv/src/kobocat/manage.py migrate --fake reversion
      • docker exec -i -t koboinstall_kobocat_1 python /srv/src/kobocat/manage.py migrate --fake oauth2_provider
      • docker exec -i -t koboinstall_kobocat_1 python /srv/src/kobocat/manage.py migrate --fake authtoken
      • docker exec -i -t koboinstall_kobocat_1 python /srv/src/kobocat/manage.py migrate --fake taggit
      • rerun docker exec -i -t koboinstall_kobocat_1 bash /srv/src/init.sh
    • docker exec -i -t koboinstall_kobocat_1 bash /srv/src/sync_static.sh

  6. docker exec into dkobo and do:

    • docker exec -i -t koboinstall_dkobo_1 bash /srv/src/init.sh -> ONLY ON NEW INSTALLS!
    • Temporary solution for postgres tables mess
      • docker exec -i -t koboinstall_dkobo_1 python /srv/src/koboform/manage.py migrate --fake hub
      • rerun docker exec -i -t koboinstall_dkobo_1 bash /srv/src/init.sh
    • docker exec -i -t koboinstall_dkobo_1 bash /srv/src/sync_static.sh
    • docker exec -i -t koboinstall_dkobo_1 bash /srv/src/create_demo_user.sh -> ONLY FOR DEV / DEMO INSTALLS.

  7. docker exec into mongo and do:

    • docker exec -i -t koboinstall_mongo_1 bash /srv/add_index.sh <- this will add a missing index to mongo db that will greatly improve speed

  8. Install enketo.conf in nginx

Testing environment

Access django admin

https://kobo.unhcr.dev/admin/

Nginx

Access nginx and restart

docker exec -i -t koboinstall_web_1 bash
nginx -s reload

nginx configuration koboinstall_web_1

  • enketo_http.conf
  • enketo_https.conf
  • enketo_include.conf

enketo .conf

Check nginx SSL certificates if you change the domain

2) if you pull the latest kobo-install, you will see 2 variables in set_vars: KOBO_SSL_KEY and KOBO_SSL_CRT you need to assign to those two the key and certificate followinng this rule: a. get rid of "---- BEGIN---" and "----- END----- " lines b. whats left, put in a single line separated by spaces. those two variables need to be enclosed by double colon.

Step Two — Configure Nginx to Use SSL

  • Login in the host and create the key and the certificate: openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout ~/src/ssl/nginx.key -out ~/src/ssl/nginx.crt

  • Set the variable in set_vars after converting \n to spaces: cat nginx.* | tr '\n' ' '

<old> * Modify ssl_certificate and ssl_certificate_key directives in kobo.conf in /etc/nginx 

sed -i 's/humanitarianresponse_info.bundle/ssl\/nginx/g' /etc/nginx/kobo.conf
sed -i 's/humanitarianresponse_info/ssl\/nginx/g' /etc/nginx/kobo.conf
</old>

nginx header

Add header

add_header X-Whom Zonker;

Check nginx upstream settings

Using the docker host interface (172.12.42.1:8000) raise an error in koboinstall_web_1 contaniner. Set the upstream IP to the container addresses as suggested in using nginx alongside the docker install:

docker inspect -f '{{ .NetworkSettings.IPAddress }}' koboinstall_dkobo_1
docker inspect -f '{{ .NetworkSettings.IPAddress }}' koboinstall_kobocat_1

upstream kobocat {
#    server 172.17.42.1:8001;
    server 172.17.0.??:8000;
}
...
upstream koboform {
    server 172.17.0.??:8000;
#    server 172.17.42.1:8000;
}

Kobocat

To restart the service 

sv restart wsgi

Kobocat settings in /srv/src/koboform.ini: 

accesslog = '/var/log/wsgi/kc.access.log'
errorlog = '/var/log/wsgi/kc.error.log'
loglevel = 'debug'

Logging in kobocat

import logging
logging.basicConfig(filename='/tmp/kc.logging.log',level=logging.DEBUG)
logging.info('*** Log init\n\n')

logging.debug('\n------------------ XForm');
logging.debug(vars(XForm));
logging.debug('------------------ /XForm\n');

Logging in kobocat (OLD)

Create the directory (mkdir -p /var/log/uwsgi) and add this in /srv/src/kobocat.ini: 

#location of log files
logto = /var/log/uwsgi/%n.log

Kobo

Fix SSL certificate check in dev

If in development server the SSL certificates are self-signed the dkobo application will raise an exception while publishing a form to a new project. From http response: 

{"status_code": 504, "detail": "[Errno 1] _ssl.c:510: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed"}
* Login in koboinstall_dkobo_1 container: docker exec -i -t koboinstall_dkobo_1 bash

  • Remove the validation check on POST request in survey_draft_views.py code
diff --git a/dkobo/koboform/views/survey_draft_views.py b/dkobo/koboform/views/survey_draft_views.py
index 7af059e..5adbd1d 100644
--- a/dkobo/koboform/views/survey_draft_views.py
+++ b/dkobo/koboform/views/survey_draft_views.py
@@ -241,7 +241,7 @@ def publish_survey_draft(request, pk, format=None):
     payload = {u'text_xls_form': valid_xlsform_csv_repr}
     try:
         url = kobocat_integration._kobocat_url('/api/v1/forms', internal=True)
-        response = requests.post(url, headers=headers, data=payload)
+        response = requests.post(url, headers=headers, data=payload, verify=False)
         status_code = response.status_code
         resp = response.json()
     except Exception, e:

~Fix OfflineGenerationError~

OfflineGenerationError at /
You have offline compression enabled but key "eeda3c82d1044337864a1ddf68860b7b" is missing from offline manifest. You may need to run "python manage.py compress".

  • Login in koboinstall_dkobo_1 container: docker exec -i -t koboinstall_dkobo_1 bash

  • Run compress in koboform src directory and resync with nginx 

    cd  /srv/src/koboform
python manage.py compress
bash /srv/src/sync_static.sh

Enketo

  • login 

    docker exec -i -t koboinstall_enketo_express_1 bash

  • logs 

    docker logs -f koboinstall_enketo_express_1

  • restart: 

    kill -2 `ps -elf | grep "\/usr\/bin\/nodejs \/srv\/enketo-express" | cut -b 10-20`

Fix SSL certificate validation

SSL certificate validatipon

Add to setup/docker/envfile.txt

# Avoid validation of SSL certificate
# http://stackoverflow.com/questions/23601989/client-certificate-validation-on-server-side-depth-zero-self-signed-cert-error
NODE_TLS_REJECT_UNAUTHORIZED=0;

Add in app.js: 

process.env.NODE_TLS_REJECT_UNAUTHORIZED = "0";


diff --git a/app/lib/communicator/communicator.js b/app/lib/communicator/communicator.js
index fc7ebe1..d291784 100644
--- a/app/lib/communicator/communicator.js
+++ b/app/lib/communicator/communicator.js
@@ -201,6 +201,8 @@ function _request( options ) {
     var error;
     var r;
     var method;
+options.rejectUnhauthorized = false; 
+options.strictSSL = false;

./app/controllers/submission-controller.js:83:options.rejectUnhauthorized = false;
./app/lib/communicator/communicator.js:144:options.rejectUnhauthorized = false; 
./app/lib/communicator/communicator.js:217:options.rejectUnhauthorized = false;

Install nginx in the host as proxy

Docker Explained: How To Containerize and Use Nginx as a Proxy

Install nginx 

sudo apt-get install -y nginx
Move old nginx.conf: sudo mv /etc/nginx/nginx.conf /etc/nginx/nginx.conf.old

Configure nginx: sudo vi /etc//etc/nginx/nginx.conf

daemon off;

worker_processes 1;

events { worker_connections 1024; }

http {

    sendfile on;

    gzip              on;
    gzip_http_version 1.0;
    gzip_proxied      any;
    gzip_min_length   500;
    gzip_disable      "MSIE [1-6]\.";
    gzip_types        text/plain text/xml text/css
                      text/comma-separated-values
                      text/javascript
                      application/x-javascript
                      application/atom+xml;

    # List of application servers
    upstream app_http_servers {
        server 172.17.42.1:80;
    }
    upstream app_https_servers  {
        server 172.17.42.1:443;
    }


    # Configuration for the server
    server {

        # Running port
        listen 10.9.153.142:80;

        # Proxying the connections connections
        location / {
            proxy_pass         http://app_http_servers;
            proxy_redirect     off;
            proxy_set_header   Host $host;
            proxy_set_header   X-Real-IP $remote_addr;
            proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header   X-Forwarded-Host $server_name;
        }
    }
    # Configuration for the server
    server {

        # Running port
        listen 10.9.153.142:443 ssl;

    ssl_certificate           /home/emav/src/ssl/nginx.crt;
    ssl_certificate_key       /home/emav/src/ssl/nginx.key;

    ssl on;
    ssl_session_cache  builtin:1000  shared:SSL:10m;
    ssl_protocols  TLSv1 TLSv1.1 TLSv1.2;
    ssl_ciphers HIGH:!aNULL:!eNULL:!EXPORT:!CAMELLIA:!DES:!MD5:!PSK:!RC4;
    ssl_prefer_server_ciphers on;

        # Proxying the connections connections
        location / {
            proxy_pass         https://app_https_servers;
            proxy_redirect     off;
            proxy_set_header   Host $host;
            proxy_set_header   X-Real-IP $remote_addr;
            proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header   X-Forwarded-Host $server_name;
        }
    }
}

Enable forwarding with UFW: sudo vi /etc/default/ufw

Scroll down and find the line beginning with DEFAULTFORWARDPOLICY.

Replace: DEFAULT_FORWARD_POLICY="DROP" With: DEFAULT_FORWARD_POLICY="ACCEPT"

Save and reload the UFW: sudo ufw reload

docker-machine

  • Install docker-machine: 

    sudo curl -L https://github.com/docker/machine/releases/download/v0.4.0/docker-machine_linux-amd64 > /usr/local/bin/docker-machine
sudo chmod +x /usr/local/bin/docker-machine

  • Create docker machine: docker-machine create --driver=virtualbox kobo-docker-machine

  • Get the IP of the host container: docker-machine ip kobo-docker-machine

  • Get the right connection information: eval "$(docker-machine env kobo-docker-machine)"

  • Edit the file, run the composer and needed commands

Docker Machine, Compose, and Swarm: How They Work Together

/misc

  • bin/dockssh 

    #!/bin/bash
docker exec -i -t $1 bash

  • autocomplete dockssh 

    # set autocomplete for docker ssh
complete -W "$(docker ps --format="{{.Names}}" | tr '\n' ' ')" dockssh

Diagram of Docker Running Containers https://docs.google.com/drawings/d/1HQC09jIVo97so-zE22yEvBKBOYfhlwR7yeCJAAa0Jzs/edit

Diagram of Docker Images (including base layers) https://docs.google.com/drawings/d/1LnaWyeC843ddFG9F0JjmjQ_BFtO8qMg2VdGYtpuOKh8/edit

Docker Image-Building Files https://bitbucket.org/teodorescuserban/kobo-dockers

Docker Hub Images

lenovo-emav certificates

$e KOBO_SSL_KEY="MIIEvwIBADANBgkqhkiG9w0BAQEFAASCBKkwggSlAgEAAoIBAQDXfhEq6SeJ8e0S 21wl6qcO96ed7/c4LioD7kXgKQHnO6Z3aQgySMgiKPXI1Nhl8Zkie08nsAIFfoPB bL3uPsMmsrjPLFMcz95qbVeriMZoQqaZhjO7U3e/HpBDaqOvOW3wkfVUHOjIc7pM vgcciVHxW6fPT3APVXpE5S3BdsJvpbiCIETbVi0czFoDB12QykKAFZebRtmFUuBf qpm7cuFNkNe9bKLiM8LgTJOIb++U60MD8UsheO0FsNMmZDLNNo1+xExY9G6BojjK P+gckT07jk+tFwWASasdgAs6L04XLBJQHBJ2wGknXLiQIX2zhQFC8JtCYJzfbBQC 1bnCSsB7AgMBAAECggEAfSzA4jE/g3ThpqN/upK86gfX+PlsnNjMFS5yroyDi4Nt 5k0/sp3ui1FJvJhO/6DAuHPyffZSpkhgVppJFbyldbQ5gDaaLNFevGm5lG3F+shE qEd5g2Yv1/f0CwVGAcIVZ7oRTQPiAJag8HUuAliOHeqE5vqY+H9Jn/2Uf53xwhHq ++HVl+AQyeXq0tvgEvm4N2q4DjmrtHvnYAiz+5pPcwxW2SEZb+n/+qCf7131J3IR obW3jbHmX5xly+tyK0pfeJSGtR/7Jd2TjyLXYpyXkwPNTTJsIT7dLasOzfJhh8X2 UZVL9MJX5LTLr7gUhEvxaKKc1mZ54xYmqAmpysMJQQKBgQD+5ZbNeimpwgt28NVm 1Xco2mHt/JkroWO7Y2MaLGXGEnMrfORxF2FP5BJsL1jbbEeHyHgvswxC6RqYnsWh TaV9l+gOt8I8rzDcmyvA9OWGIknf8I26RAEUqucC2Gjfme74b8O6Jo3IdobIao0+ 3zcLGuGBZzP97gQ29yFdc/WOoQKBgQDYbNH5jpS08TLrHzkJ3vyZltWGTyXnFsuQ vcsiAPAFoQW73sh6aVbAEn0FCELD4lUBbqbi/R48ErW7FRjossjWewEEDIsgQ9sH MXaZGc4CdSL/3ZHOGVsZBeELLDEnrz1elAejmFuW+uVqYIB66wfb94U8dEx8qewx RvW36axFmwKBgQC/k/2w+DViMsb9IWuzO5Ow79oUzfqtkUJtL/jnTiE8KtfHArqL yV/JdQjnqPpnCM4GXwT+DNuVWrAzr3moj7xnVgoDLdfeVmoqFLJg92+fThwqJRnQ upX5W63/NHo2PPMgZJg+TAXaekM5VOROMeSkAxMkeFXab89ZJIs+Ow5/oQKBgQCO jves29dmbK6uo61trB1Vij5yLvZ3htNmLy574oQ4IP3m53PE9lg3QrZpNyzLXO3D eMCjKjndNjHQ3q++r+2NIg7kPOLzyN6gmVtWs+G4SrJYOfnN+XPBQtcfBegHDO5u uzFg+H30MKYncROQY/qUUTHJRY4QW9mCCAzZLSnV2wKBgQCEsSnT2T80j8Yd7TD8 PQK/fHvfaqAb99H3XL1MLXo9lLHuxCP4ClY+96GlclWhn2gZT/hj/wdJBUZEzAav qN8CQApkNlEdkgjavlFoamv1idCqQ37WM1QuzBx2marrSC4gjvhz46k/c7ZOv3Fs GTZZzAQd/I9XVrUUTDQwLBIIUg=="
$e KOBO_SSL_CRT="MIIDXTCCAkWgAwIBAgIJAPHrtUu6JN3RMA0GCSqGSIb3DQEBCwUAMEUxCzAJBgNV BAYTAkFVMRMwEQYDVQQIDApTb21lLVN0YXRlMSEwHwYDVQQKDBhJbnRlcm5ldCBX aWRnaXRzIFB0eSBMdGQwHhcNMTUxMTE4MTM0NjI4WhcNMTYxMTE3MTM0NjI4WjBF MQswCQYDVQQGEwJBVTETMBEGA1UECAwKU29tZS1TdGF0ZTEhMB8GA1UECgwYSW50 ZXJuZXQgV2lkZ2l0cyBQdHkgTHRkMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIB CgKCAQEA134RKuknifHtEttcJeqnDvenne/3OC4qA+5F4CkB5zumd2kIMkjIIij1 yNTYZfGZIntPJ7ACBX6DwWy97j7DJrK4zyxTHM/eam1Xq4jGaEKmmYYzu1N3vx6Q Q2qjrzlt8JH1VBzoyHO6TL4HHIlR8Vunz09wD1V6ROUtwXbCb6W4giBE21YtHMxa AwddkMpCgBWXm0bZhVLgX6qZu3LhTZDXvWyi4jPC4EyTiG/vlOtDA/FLIXjtBbDT JmQyzTaNfsRMWPRugaI4yj/oHJE9O45PrRcFgEmrHYALOi9OFywSUBwSdsBpJ1y4 kCF9s4UBQvCbQmCc32wUAtW5wkrAewIDAQABo1AwTjAdBgNVHQ4EFgQUQw9APkhj Xus7zQY9nszf7bqSrrQwHwYDVR0jBBgwFoAUQw9APkhjXus7zQY9nszf7bqSrrQw DAYDVR0TBAUwAwEB/zANBgkqhkiG9w0BAQsFAAOCAQEAsjOiGo7yg9sg6JjLW1zG z8Vvyf3BbJ4R4emxXxQxo56+Dc0A1vqF0FF4TU0UaVhXhZZHc7zZZdXkSAaLLo2/ mNJMvEnNZqG/NHqIkwaeIhLVMoID2AvMAhglpIk6RUAlf4bwEoZg/0e2IuGZnxUd z2fcDjBvDihfbtB3P8LQzrj+3ZuJ1fVhpwfpaVGeeCxeNtYUvXXyFAu9S8QgEhid dM62IzWl29nsKrKOujUChPoRXJubZ8vUsm7i6gdTipO52jU37sb+Qdz0nKSA83Fj V4asnbeKP3dWQPe2o+5663h1auQp6mcCSsc676L7tz/hwsjPo8KMvbCoOXj9T2F7 xg=="

Welcome

Welcome to your wiki! This is the default page we've installed for your convenience. Go ahead and edit it.

Wiki features

This wiki uses the Markdown syntax.

The wiki itself is actually a git repository, which means you can clone it, edit it locally/offline, add images or any other file type, and push it back to us. It will be live immediately.

Go ahead and try:

$ git clone https://unhcr@bitbucket.org/unhcr/kobo-install.git/wiki

Wiki pages are normal files, with the .md extension. You can edit them locally, as well as creating new ones.

Syntax highlighting

You can also highlight snippets of text (we use the excellent Pygments library).

Here's an example of some Python code:

#!python

def wiki_rocks(text):
    formatter = lambda t: "funky"+t
    return formatter(text)

You can check out the source of this page to see how that's done, and make sure to bookmark the vast library of Pygment lexers, we accept the 'short name' or the 'mimetype' of anything in there.

Have fun!

Updated