1. OpenREM Avatar OpenREM
  2. OpenREM
  3. OpenREM
  4. Issues

Docs for Docker

Issue #823 new
Ed McDonagh created an issue

Need to add docs for Docker, push to the preferred install mechanism, with pip/manual on linux as an ‘allowed’ alternative.

Comments (135)

  29. Ed McDonagh reporter

    Updating code_dev_env.rst with some things from startservices and some ideas from the wiki page about Windows dev env. Refs #823 [skip ci] docs only.

    → <<cset 1a67060a9b3d>>

  30. Ed McDonagh reporter

    Working on removing start services docs, adding concurrency instruction here for now; might be better to remove as default and have instructions in docs. Refs #823 [skip ci] docs only.

    → <<cset 4651972ec3eb>>

  32. Ed McDonagh reporter

    Set concurrency to default in both Docker and Linux installs, new doc to explain how to specify. Next is to add Flower options. Refs #823 [skip ci] docs only

    → <<cset c71c488f5d18>>

  33. David Platten

    I’m just at the start of following the docs. I’m going to post questions here as I go along.

    The first relates to changing DJANGO_ALLOWED_HOSTS in the .env.prod file, https://docs.openrem.org/en/latest/env_variables.html

    I know I need to include 127.0.0.1, but I don’t know what the [::1] means, and I don’t know if I have to leave the existing openrem and nginx there, or replace them with something else.

  34. Ed McDonagh reporter

    [::1] is local host in IPv6.

    Having more than you need in there is fine. I agree that the text could be more helpful. What you need (I think) is the servername that clients will be using, but I need to check that it doesn’t need the container name for the in-Docker networking.

    I’ll do some testing.

  35. David Platten

    Running docker-compose up -d results in the following error on my Windows 10 Pro laptop:

    D:\docker\OpenREM1.0dev>docker-compose up -d
    openrem10dev_broker_1 is up-to-date
    openrem-orthanc-1 is up-to-date
    openrem-db is up-to-date
    openrem10dev_worker_1 is up-to-date
    openrem is up-to-date
    openrem10dev_flower_1 is up-to-dateStarting openrem-nginx ...
    Starting openrem-nginx ... error
    ERROR: for openrem-nginx Cannot start service nginx: driver failed programming external connectivity on endpoint openrem-nginx (7df2d279e87d0ba75a56620ca149709c786d3c6afd9738213181520800f1b91f): Error starting userland proxy: listen tcp 0.0.0.0:80: bind: An attempt was made to access a socket in a way forbidden by its access permissions.

    ERROR: for nginx Cannot start service nginx: driver failed programming external connectivity on endpoint openrem-nginx (7df2d279e87d0ba75a56620ca149709c786d3c6afd9738213181520800f1b91f): Error starting userland proxy: listen tcp 0.0.0.0:80: bind: An attempt was made to access a socket in a way forbidden by its access permissions.
    ERROR: Encountered errors while bringing up the project.

    I’ve cleaned out my other Docker containers and images, and still get this error.

    ‌

  36. Ed McDonagh reporter

    What happens if you change it to a high port - in docker-compose.yml in the nginx section change the ports from 80:80 to 8080:80

  37. Ed McDonagh reporter

    Did it let you bring Orthanc up on port 104? In which case I wonder if you are running IIS or something, and the port is already in use. There’ll be some sort of netstat command you can run to see.

  38. David Platten

    I’ve run a netstat command and found the PID of the process on port 80, but it’s not IIS. It’s a system process that I can’t track down (netstat -n -a -o). It’s ntoskrnl.exe that is using the port, located in C:\Windows\System32.

    Changing this bit of the docker-compose.yml from

    ports:
  - 80:80

    to

    ports:
  - 8080:80

    Works.

  39. Ed McDonagh reporter

    And presumably as it was a clash rather than a high-port/low-port thing, it might work on 81 or similar too?

  42. David Platten

    I think the command should be:

    docker cp /path/to/openremdump.bak openrem-db:db_backup/

    (at least, that seemed to work)

  43. Ed McDonagh reporter

    Oh no! There shouldn’t be docker at the start of that! You are just copying into the local folder

  44. David Platten

    When I ran the

    docker-compose exec openrem python manage.py makemigrations remapp

    I was asked:

    Did you rename userprofile.median_available to userprofile.plotBoxplots (a BooleanField)? [y/N] n
    Did you rename userprofile.median_available to userprofile.plotMedian (a BooleanField)? [y/N] n

    I answered n to both

  45. David Platten

    When I ran the command:

    docker-compose exec openrem python django-admin compilemessages

    I received the following error:

    python: can't open file 'django-admin': [Errno 2] No such file or directory

  46. David Platten

    Despite the django-admin error I do now seem to have a working Docker-based installation which is using the migrated 0.10.0 database. Hooray!

  51. David Platten

    @Ed McDonagh my Orthanc container won’t start. How do I trouble-shoot it?

    PS D:\docker\OpenREM1.0dev> docker container list
CONTAINER ID        IMAGE                          COMMAND                  CREATED             STATUS                            PORTS                                                                  NAMES
158b449c2f5d        nginx:1.17.8-alpine            "nginx -g 'daemon of…"   2 days ago          Up 45 hours                       0.0.0.0:8080->80/tcp                                                   openrem-nginx
af13d80272e1        openrem/openrem:develop        "/home/app/openrem/e…"   3 days ago          Up 45 hours                       8000/tcp                                                               openrem
784f83f47c3f        openrem/openrem:develop        "/home/app/openrem/e…"   3 days ago          Up 45 hours                       5555/tcp                                                               openrem10dev_flower_1
975c6a1415d2        openrem/openrem:develop        "/home/app/openrem/e…"   3 days ago          Up 45 hours                                                                                              openrem10dev_worker_1
b69695875b0b        openrem/orthanc                "/bin/bash -c 'apt-g…"   3 days ago          Restarting (100) 19 seconds ago                                                                          openrem-orthanc-1
5a74ce55d787        postgres:12.0-alpine           "docker-entrypoint.s…"   3 days ago          Up 45 hours                       5432/tcp                                                               openrem-db
70aca4af0b1c        rabbitmq:3-management-alpine   "docker-entrypoint.s…"   3 days ago          Up 45 hours                       4369/tcp, 5671-5672/tcp, 15671-15672/tcp, 15691-15692/tcp, 25672/tcp   openrem10dev_broker_1
PS D:\docker\OpenREM1.0dev>

    ‌

  53. Ed McDonagh reporter

    I need to do all the troubleshooting docs.

    List containers with docker-compose ps

    Get logs for the orthanc container with docker-compose logs -f orthanc_1 see if that gives you a clue.

    The name we use is that of the service we define in the docker-compose.yml file, so orthanc_1.

  54. David Platten

    Swapping

    image: openrem/orthanc

    for the vanilla

    image: osimis/orthanc

    in the orthanc_1 section of docker-compose.yml has made it work for me. It must be related to the package update and installation in the openrem/orthanc container. However, the logs indicate that the updating had worked.

    ‌

  55. David Platten

    Actually, the Dockerfile contents for openrem/orthanc is:

    FROM osimis/orthanc:master
RUN apt-get update && apt-get -y install zip unzip

    It doesn’t actually tell Orthanc to run, it just runs an apt update and install.

    Should there be a CMD in there at the end to specify which command to run within the container?

  56. Ed McDonagh reporter

    Assuming your openrem/orthanc image wasn’t old, it shouldn’t have made any difference. Can you see how old that image is (docker images) - and if it is more than 14 days old can you pull a new one and try it again? (docker pull openrem/orthanc)

    The Dockerfile simply adds zip and unzip to the main release. I’m considering removing the functions so it isn’t necessary!

  57. Ed McDonagh reporter

    Hi David. Can you pull latest in again, and see if it now works for you? (docker pull openrem/orthanc:latest)

    I have triggered a new build, and presumably the Osimis image has been updated because it only used the first two layers from cache. I have no idea what went wrong with the last one, obviously something did!

    It now works for me again, specifying openrem/orthanc:latest in the docker-compose.yml file.

  59. David Platten

    I’ve set up a test Docker-based OpenREM system on a new server running Windows Server 2019.

    At the moment I have installed Docker Desktop.

    I’ve created a scheduled task to start Docker Desktop on boot. At the moment this is running as my local user - I need to get it running using the SYSTEM account.

    When the system reboots OpenREM comes back to life. However, Celery does not. This is because the celery.pid file still exists in the logs folder.

    Is there scope to delete celery.pid as part of the start up of OpenREM?

  60. David Platten

    I have written a basic batch file that deletes logs\celery.pid before Docker Desktop is started. Hopefully this will fix the problem for me.

  61. Ed McDonagh reporter

    I thought I had dealt with that, but I obviously haven’t.

    Can you disable the batch file that deletes the pid file, and modify the docker-compose.yml file as follows:

    worker:
    command: celery worker -A openremproject -Q default --logfile=/logs/celery.log

    i.e. to remove the --pidfile flag.

    Then try again 🙂

  62. David Platten

    @Ed McDonagh that worked - thanks. I just have to work out how to get Docker to launch at boot as the SYSTEM user rather than as me (my password will change at some point).

    I also think that I should be using Docker Enterprise instead of Docker Desktop - that’s probably the answer.

  63. David Platten

    For my Windows Server 2019 Docker installation port 80 worked for the webserver:

    ports:
  - 80:80

    ‌

  68. David Platten

    On Linux there’s a permissions issue once you’ve extracted the develop.zip file. When I tried the docker-compose up -d command several of the services kept on restarting. I then made some very generous changes to the permissions of the folder and all files in it, and the docker-compose up -d worked OK. Not sure what the exact permission requirements are.

    ‌

    ‌

  70. David Platten

    I created a new Ubuntu user called openrem and added it to the docker and sudo groups (I had to create the docker group first). I then logged in as this user and downloaded and extracted the develop.zip file into the openrem user's home directory.

  71. David Platten

    I perhaps should add that I installed Docker during the Ubuntu Server installation process as a snap.

  72. Ed McDonagh reporter

    That is probably relevant, as I don’t think I’ve seen this before. But snap changes everything!

  73. David Platten

    I just thought that using the snap was the easiest way of installing Docker, especially when it was being offered as part of the operating system process.

  74. Ed McDonagh reporter

    Absolutely. And we should definitely try it. But we should expect some permission issues etc as it is effectively a locked down container itself!

  75. David Platten

    If I add a user: root line in each service then I don't have any permissions errors when running the snap-based Docker installation. Extract below:

    ‌

    services:
  openrem:
    container_name: openrem
    user: root
    restart: unless-stopped
    image: openrem/openrem:develop

    The new line is required for each service in the file.

    Without this, the containers are being run by a user called app which doesn’t have any rights to change files in the host folders.

    When using the above method the files that are created in the bind mounted folders are owned by root. I don’t know if that is a problem:

    openrem@openremubuntu:~/openrem-docker-6dda9460edd3$ ls -al ./logs/
total 12
drwxrwxr-x 2 openrem openrem 4096 Jan 28 16:56 .
drwxrwxr-x 7 openrem openrem 4096 Jan 28 16:52 ..
-rw-r--r-- 1 root    root     772 Jan 28 16:53 celery.log
-rw-rw-r-- 1 openrem openrem    0 Jan 15 22:55 .gitkeep
-rw-r--r-- 1 root    root       0 Jan 28 16:52 openrem_extractor.log
-rw-r--r-- 1 root    root       0 Jan 28 16:52 openrem.log
-rw-r--r-- 1 root    root       0 Jan 28 16:52 openrem_qr.log
-rw-r--r-- 1 root    root       0 Jan 28 16:52 openrem_store.log

    ‌

  76. Ed McDonagh reporter

    With apt installed docker, and no user defined in docker-compose.yml:

    mcdonaghe@FRP5019Ubuntu:~/research/test-docker/openrem-docker-ffa6c5d33790$ ls -l logs/
total 96
-rw-r--r-- 1 mcdonaghe mcdonaghe 27976 Jan 28 08:58 celery.log
-rw-r--r-- 1 mcdonaghe mcdonaghe     2 Jan  4 21:12 celery.pid
-rw-r--r-- 1 mcdonaghe mcdonaghe     0 Dec 29 12:40 openrem_extractor.log
-rw-r--r-- 1 mcdonaghe mcdonaghe 54138 Jan 19 08:59 openrem.log
-rw-r--r-- 1 mcdonaghe mcdonaghe     0 Dec 29 12:40 openrem_qr.log
-rw-r--r-- 1 mcdonaghe mcdonaghe     0 Dec 29 12:40 openrem_store.log

mcdonaghe@FRP5019Ubuntu:~/research/test-docker/openrem-docker-ffa6c5d33790$ docker-compose exec openrem ls -l /logs
total 96
-rw-r--r-- 1 app app 27976 Jan 28 08:58 celery.log
-rw-r--r-- 1 app app     2 Jan  4 21:12 celery.pid
-rw-r--r-- 1 app app 54138 Jan 19 08:59 openrem.log
-rw-r--r-- 1 app app     0 Dec 29 12:40 openrem_extractor.log
-rw-r--r-- 1 app app     0 Dec 29 12:40 openrem_qr.log
-rw-r--r-- 1 app app     0 Dec 29 12:40 openrem_store.log

    So in Docker, the user is app, the same files in Ubuntu are owned by me.

    Having them owned by root might cause an issue, because the docker build assumes Django and the python files are owned by app

  77. David Platten

    Maybe we should require the use of the non-snap version of Docker. There are lots of discussions online about permission issues with the snap version.

  78. David Platten

    I have now removed the snap-based Docker and am using an apt-installed one from the Docker repository following the instructions here: https://docs.docker.com/engine/install/ubuntu/

    I am running into permission issues with this too when running the docker-compose exec openrem python manage.py migrate remapp --fake command.

    The “app” user doesn’t have permission to write files to the logs folder on the host.

    ‌

  79. Ed McDonagh reporter

    Is this with a fresh folder - you don’t have any if the previous attempts files or permissions in there?

  80. David Platten

    Yes, a fresh folder - I deleted the old one entirely, and then unzipped the develop.zip file again.

  81. David Platten

    The permissions issue is specific to the logs folder. Allowing write access to all users from the host (sudo chmod 777 ./logs) fixes the problem for the snap and apt versions of Docker.

  82. Ed McDonagh reporter

    And which user owns the files on the Ubuntu side (and which on the Docker container side)?

  83. David Platten

    On the host side:

    openrem@openremubuntu:~/openrem-docker-6dda9460edd3$ ls -al ./logs/
total 12
drwxrwxrwx 2 openrem  openrem  4096 Jan 29 10:35 .
drwxrwxr-x 7 openrem  openrem  4096 Jan 29 10:23 ..
-rw-r--r-- 1 dplatten dplatten  708 Jan 29 10:23 celery.log
-rw-rw-r-- 1 openrem  openrem     0 Jan 15 22:55 .gitkeep
-rw-r--r-- 1 dplatten dplatten    0 Jan 29 10:19 openrem_extractor.log
-rw-r--r-- 1 dplatten dplatten    0 Jan 29 10:19 openrem.log
-rw-r--r-- 1 dplatten dplatten    0 Jan 29 10:19 openrem_qr.log
-rw-r--r-- 1 dplatten dplatten    0 Jan 29 10:19 openrem_store.log
-rw-r--r-- 1 dplatten dplatten    0 Jan 29 10:35 testing

    On the container side:

    openrem@openremubuntu:~/openrem-docker-6dda9460edd3$ docker-compose exec openrem ls -al /logs
total 12
drwxrwxrwx 2 1001 1001 4096 Jan 29 10:35 .
drwxr-xr-x 1 root root 4096 Jan 29 10:23 ..
-rw-rw-r-- 1 1001 1001    0 Jan 15 22:55 .gitkeep
-rw-r--r-- 1 app  app   708 Jan 29 10:23 celery.log
-rw-r--r-- 1 app  app     0 Jan 29 10:19 openrem.log
-rw-r--r-- 1 app  app     0 Jan 29 10:19 openrem_extractor.log
-rw-r--r-- 1 app  app     0 Jan 29 10:19 openrem_qr.log
-rw-r--r-- 1 app  app     0 Jan 29 10:19 openrem_store.log
-rw-r--r-- 1 app  app     0 Jan 29 10:35 testing

    Hmm. I was expecting the host side to be owned by the openrem user.

  84. Ed McDonagh reporter

    So that is the same as mine now (apart from the openrem user) Was the 777 required? Or was that because the folder is owned by openrem and the files are written by dplatten?

  85. David Platten

    I had to set 777 for the logs folder for it to work. I don’t understand why dplatten is involved. I may reset things and try again.

  87. David Platten

    I’ve just purged all of my containers and volumes, deleted the unzipped folder and started again.

    All using the openrem user.

    I had to use sudo chmod 777 ./logs again - after the initial docker-compose up -d several of the containers were in a restarting loop.

    Once I’d made the chmod change the containers all came up.

    However, on the host the log files are all owned by dplatten, not by openrem.

  88. David Platten

    @Ed McDonagh does bringing the logs, db_backup etc folders into the containers require a rebuild of the docker images?

  89. Ed McDonagh reporter

    Are you referring to implementing the decision we made to not use bind mounts?

    I’ve not worked it all through yet. For containers like nginx, using bind mounts is really useful to add configurations to vanilla upstream images. And for database backups and logs the tools aren’t quite as nice as I’d hoped - there is a cp function in either direction, which I guess could work for the config too, but no rm or similar.

    So we might do something like:

    Create db backups as before (we are referencing the container internal path as before, but this time it would be in a volume):

    docker-compose -f /path/to/docker-compose.yml exec db pg_dump -U openrem_user -d openrem_prod -F c -f "/db_backup/openremdump-"$TODAY"

    But then you copy it out using 

    docker cp openrem:db_backup/openremdump* .

    Looking at the logs might be:

    docker-compose exec openrem ls -lrth /logs
docker-compose exec openrem less /logs/openrem-qr.log

    I haven’t tried these, so I’ve probably got the syntax wrong. Just wanted to get this message written before going out for a walk - this is the third attempt I’ve had to get this written - keep getting interrupted and losing the text!

    ‌

  91. Ed McDonagh reporter

    @David Platten I’ve had a stab at a docker-compose.yml file and docs that don’t have any bind volumes or secrets files.

    Download https://bitbucket.org/openrem/docker/get/nobind.zip

    Follow the install or upgrade docs at https://docs.openrem.org/en/issue823nobindsecrets/installation.html and https://docs.openrem.org/en/issue823nobindsecrets/release-1.0.0.html#set-up-the-new-installation

    I haven’t had a chance to test it, so expect mistakes. If you do, let me know!

  93. Ed McDonagh reporter

    Having the Orthanc scripts folder as a non-bind volume is a bit of a pain it turns out. Because the Lua script is referenced in the Orthanc config, the container falls over on startup. But we can’t copy the script in until the container is up.

    So we could work around it with a temporary container like one of these examples: https://github.com/moby/moby/issues/25245

    Or we could use bind folders for things that the containers will read but not write - so Orthanc and Nginx configs in bind mounts, and things that the container will write such as logs, we use a non-bind volume.

    What do you think @David Platten ?

  94. David Platten

    I agree@Ed McDonagh - I think that using bind folders where the contents will be read is a good idea. The folders that need to be written to by containers can be kept inside the container.

  101. David Platten

    I’m trying the new version out now.

    Copying the database backup to the container folder needs to use “openrem-db”, not “db”:

     docker cp /home/openrem/2021-01-25_1627_OpenREMdatabase.backup openrem-db:/db_backup/

    ‌

  103. Ed McDonagh reporter

    You are right. For docker-compose we have to use the service name defined in docker-compose.yml (db in this example), whereas for docker we need to use the container name, (openrem-db here).

    What is troubling me is that I don’t know how all the containers get the prefix openrem- - will that always be the case?

    Some of the containers are named with a prefix of the folder name (like openrem-docker-f4b53e8154b1_broker_1) but others don’t and I don’t know why. When I tried to start a second folder’s docker-compose the broker image came up because it has the folder name in the name, but others didn’t because of the name conflict.

    We could possibly have the instruction as you had, but have a note that if it doesn’t work to check the name of the container?

  104. David Platten

    I’ve only seen the db container as openrem-db. It’s only the worker and broker that I’ve seen with the extra characters in their container names.

    ‌

    On another note, I rebooted my virtualbox-based Ubuntu install with Docker-based OpenREM still running to see what would happen. The system wouldn’t shutdown as it was waiting for postgres to shutdown, but postgres was steadfastly staying put. I think postgres was waiting for all database connections to timeout.

    Maybe the solution to this is to suggest to users that they include a script that is run at shutdown or reboot that runs docker-compose down, and also have docker-compose up -d run at boot?

  107. Ed McDonagh reporter

    Of course - how did I miss that! I’ll add them in next time I’m changing it, to tidy up. And I’ll test that you can still scale up the worker when it has a container name.

  108. David Platten

    Re Postgres blocking the reboot: I forced the virtual machine off, and then restarted it. OpenREM didn’t come back up on reboot. A docker-compose up -d gave an error. However, a docker-compose down followed by a docker-compose up -d worked.

  109. Ed McDonagh reporter

    Adding a container name prevents scaling of containers. I think the only container you would scale is the worker container - would you agree?

    I presume if you knew what you were doing there may be a case for scaling the database, but it is not simple and if you wanted to do that you’d probably be ok adapting the instructions to do so!

    This would leave just the worker with a name prefixed with the folder name.

  112. David Platten

    Started to update the system diagram to reflect the new Docker configuration. Needs further work - comments very welcome [skip ci]. Refs issue #823

    → <<cset 98fcd84e7dcd>>

  122. Ed McDonagh reporter

    Adding notes to the top of each page to warn against updating the translations as the docs are too much in flux right now. Refs #823 [skip ci] docs only

    → <<cset fbac04e422da>>

  123. David Platten

    I’ve been installing a docker-based OpenREM on a system where internet access has been removed once the operating system and the Docker snap has been installed.

    I obtained a list of required Docker images by looking in the docker-compose file.

    I then downloaded the required Docker images on a computer with internet access using these instructions:

    https://serverfault.com/questions/701248/downloading-docker-image-for-transfer-to-non-internet-connected-machine

    I copied them onto the system without internet access and ran the docker-compose up -d command. OpenREM came up without problem.

  126. Ed McDonagh reporter

    Suggestion to have the container name as the blue text, and docker container/volume as a 'type'. Refs #823. Would like to ditch serif font too...

    → <<cset 6ba917acf933>>

  136. Log in to comment
Assignee
–
Type
task
Priority
critical
Status
new
Component
Documentation
Milestone
1.0.0
Votes
0
Watchers
1
Jira Software: the preferred issue tracker for Bitbucket. Join the team!