go-docker: Swarm problem 'Connection aborted.' BadStatusLine('\x15\x03\x01\x00\x02\x02\x16',)

[Ruben]

Hi,

After resolve the issue #39, we found this error launching jobs:

2016-08-02 18:27:26,439 ERROR [godocker-scheduler][MainThread] Execute:Job:3:('Connection aborted.', BadStatusLine('\x15\x03\x01\x00\x02\x02\x16',))

If we launch the command "docker -H 192.168.99.100:3376 info", we get this message:

root@ru-Latitude-E5540:/home/ru/Escritorio/Github/Go-Docker/go-docker# docker -H 192.168.99.100:3376 info
Containers: 17
 Running: 2
 Paused: 0
 Stopped: 15
Images: 4
Server Version: swarm/1.2.4
Role: primary
Strategy: spread
Filters: health, port, containerslots, dependency, affinity, constraint
Nodes: 2
 agent1: 192.168.99.101:2376
  └ ID: 2HTQ:4BAI:XJ7U:FTOI:EANZ:GRA4:4KPG:ASMH:ZI7M:GPC6:24WE:XKW3
  └ Status: Healthy
  └ Containers: 9 (1 Running, 0 Paused, 8 Stopped)
  └ Reserved CPUs: 0 / 1
  └ Reserved Memory: 0 B / 1.021 GiB
  └ Labels: kernelversion=4.4.16-boot2docker, operatingsystem=Boot2Docker 1.12.0 (TCL 7.2); HEAD : e030bab - Fri Jul 29 00:29:14 UTC 2016, provider=virtualbox, storagedriver=aufs
  └ UpdatedAt: 2016-08-03T08:36:44Z
  └ ServerVersion: 1.12.0
 agent2: 192.168.99.102:2376
  └ ID: LQJP:2IH5:A6GA:SBJW:NDZD:LXNH:OZ4F:2BPH:ARKM:4ZXA:VG26:BEUL
  └ Status: Healthy
  └ Containers: 8 (1 Running, 0 Paused, 7 Stopped)
  └ Reserved CPUs: 0 / 1
  └ Reserved Memory: 0 B / 1.021 GiB
  └ Labels: kernelversion=4.4.16-boot2docker, operatingsystem=Boot2Docker 1.12.0 (TCL 7.2); HEAD : e030bab - Fri Jul 29 00:29:14 UTC 2016, provider=virtualbox, storagedriver=aufs
  └ UpdatedAt: 2016-08-03T08:36:58Z
  └ ServerVersion: 1.12.0
Plugins:
 Volume: 
 Network: 
Swarm: 
 NodeID: 
 Is Manager: false
 Node Address: 
Security Options:
Kernel Version: 4.4.16-boot2docker
Operating System: linux
Architecture: amd64
CPUs: 2
Total Memory: 2.042 GiB
Name: 689c244f822d
Docker Root Dir: 
Debug Mode (client): false
Debug Mode (server): false
WARNING: No kernel memory limit support

If we launch this other command "docker -H 127.0.0.1:2375 info" or "docker -H :2375 info":

root@ru-Latitude-E5540:/home/ru/Escritorio/Github/Go-Docker/go-docker# docker -H :2375 info
Cannot connect to the Docker daemon. Is the docker daemon running on this host?

and with "docker info" this other info:

root@ru-Latitude-E5540:/home/ru/Escritorio/Github/Go-Docker/go-docker# docker info
Containers: 12
 Running: 1
 Paused: 0
 Stopped: 11
Images: 2
Server Version: 1.12.0
Storage Driver: aufs
 Root Dir: /mnt/sda1/var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 28
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: host bridge null overlay
Swarm: active
 NodeID: 4wbdner4v2y60d35w4ck794vd
 Is Manager: true
 ClusterID: 27gzrg2wuqeymz80mw1znr7r4
 Managers: 1
 Nodes: 1
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot interval: 10000
  Heartbeat tick: 1
  Election tick: 3
 Dispatcher:
  Heartbeat period: 5 seconds
 CA configuration:
  Expiry duration: 3 months
 Node Address: 192.168.99.100
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 4.4.16-boot2docker
Operating System: Boot2Docker 1.12.0 (TCL 7.2); HEAD : e030bab - Fri Jul 29 00:29:14 UTC 2016
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 995.9 MiB
Name: manager
ID: 5GB2:CENI:FCJB:5N2G:SUOU:GE5D:I2NS:SE7V:ED33:5IAL:SKYF:RNS6
Docker Root Dir: /mnt/sda1/var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 33
 Goroutines: 119
 System Time: 2016-08-03T08:40:11.504711776Z
 EventsListeners: 0
Registry: https://index.docker.io/v1/
Labels:
 provider=virtualbox
Insecure Registries:
 127.0.0.0/8

How can we test go-docker with docker swarm directly?

Thanks for all your help.

[Olivier Sallou]

Do you use python 2 or 3? Is godocker scheduler installed on the same server than swarm?

[Olivier Sallou]

I see your swarm was installed with boot2docker so i suppose it runs in a VM. Did you try the "docker -H 192.xxxxx info" from the server where is godocker?

[Olivier Sallou]

At last, did you enable tls on docker or swarm? This error looks like tls issue.

[Olivier Sallou]

boot2docker seems to enable tls by default and godocker does not manage yet tls setup for docker.

I gonna try to add tls mngt to fix this.

[Ruben]

We are using Python on version 2.7

root@ru-Latitude-E5540:/home/ru/Escritorio/Github/Go-Docker/go-docker# python
Python 2.7.10 (default, Oct 14 2015, 16:09:02) 
[GCC 5.2.1 20151010] on linux2
Type "help", "copyright", "credits" or "license" for more information.

Yes, we are trying go-docker in only one machine. For doing that we are using docker swarm in sandbox like is said on this link: Swarm in a sandbox thats why you can see boot2docker on each virtual docker machine:

root@ru-Latitude-E5540:/home/ru/Escritorio/Github/Go-Docker/go-docker# docker-machine ls
NAME      ACTIVE   DRIVER       STATE     URL                         SWARM   DOCKER    ERRORS
agent1    -        virtualbox   Running   tcp://192.168.99.101:2376           v1.12.0   
agent2    -        virtualbox   Running   tcp://192.168.99.102:2376           v1.12.0   
manager   *        virtualbox   Running   tcp://192.168.99.100:2376           v1.12.0

We can run docker images on the swarm doing this: docker -H 192.168.99.100:3376 run hello-world and as you can see, each time its executed on different machine:

docker -H 192.168.99.100:3376 ps -a
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS                      PORTS               NAMES
17dd4c3630fe        hello-world         "/hello"                 3 seconds ago       Exited (0) 2 seconds ago                        agent1/thirsty_swanson
1ee72f38b8ed        hello-world         "/hello"                 5 seconds ago       Exited (0) 3 seconds ago                        agent2/drunk_tesla
558e88f54f58        hello-world         "/hello"                 14 seconds ago      Exited (0) 13 seconds ago                       agent1/prickly_kalam
8ab3e5c69eee        hello-world         "/hello"                 15 seconds ago      Exited (0) 14 seconds ago                       agent2/stupefied_dijkstra
8a5aaf981191        hello-world         "/hello"                 16 hours ago        Exited (0) 16 hours ago                         agent1/gloomy_yalow

We hay go-docker executed directly on the local machine:

root@ru-Latitude-E5540:/home/ru/Escritorio/Github/Go-Docker/go-docker# ps -ef | grep py
root      7580  1435  0 ago02 ?        00:00:01 /usr/bin/python /usr/local/bin/gunicorn -D -p godweb.pid --log-config=production.ini --paste production.ini
root      7585  7580  0 ago02 ?        00:00:13 /usr/bin/python /usr/local/bin/gunicorn -D -p godweb.pid --log-config=production.ini --paste production.ini
root      7586  7580  0 ago02 ?        00:00:13 /usr/bin/python /usr/local/bin/gunicorn -D -p godweb.pid --log-config=production.ini --paste production.ini
root     11494  1435  0 ago02 ?        00:00:27 python go-d-scheduler.py start
root     11513  1435  0 ago02 ?        00:00:07 python go-d-watcher.py start

Thanks.

[Olivier Sallou]

Closes #40, manage tls with docker daemon, impact go-d.ini with new docker section

→ <<cset bcd2dd7f2a4e>>

[Olivier Sallou]

Commit fixes the issue by enabling tls support. In go-d.ini, a new docker section is added (see go-d.ini.sample) and need to be configured to match docker daemon setup (ca cert verification, ...)

Example:

docker:
    # tcp://127.0.0.1:2375 if not tls, else https://127.0.0.1:2375
    url: 'https://127.0.0.1:2375'
    api_version: 'auto'
    tls: True
    # Authenticate server based on given CA => '/path/to/ca.pem'
    ca_cert: null
    # Authenticate with client certificate => '/path/to/client-cert.pem', '/path/to/client-key.pem'
    client_cert: '/home/osallou/.docker/cert.pem'
    client_key: '/home/osallou/.docker/key.pem'

More info: https://docker-py.readthedocs.io/en/stable/tls/

[Olivier Sallou]

you need to know how swarm is configured, you can check daemon options for that (tls / tlsverify, cacert etc....). According to setup, you need the ca/client keys and set the path to them.

Disabling tls is easier setup... ;-)