1. Kieran Evans
  2. boatload

Source

boatload /

Filename Size Date modified Message
Docker
boatload
data
fleetmanager
satellite
static/images
templates/fleetmanager/units
10 B
571 B
1.7 KB
6.8 KB
91 B
20 B
489 B
251 B
264 B
44 B
628 B

BoatLoad BoatLoad

This project consists of two parts. The API, and the Satellites. It is currently being written for a specific purpose, but may expand to be more generic in the future.

API is created using Django REST framework and uses session based auth, or JSON Web Tokens (JWT) for authentication, via djangorestframework-jwt.

Note

Boatload containers have been bumped back from Python 3.5 to 2.7 due to supervisor not yet supporting python3.

Satellites

Satellites are deployed to Fleet as Global units, and so run on all members of the CoreOS cluster. The satellites are responsible for any Docker operations not supported by the Fleet API (such as getting diffs on the container, downloading files, stdout, etc). The Satellite consists of a minimal Django app that talks to the docker API on it's hosts, and exposes a simple REST-like API. This API should not be used directly, rather should be accessed via the main API.

API

The API is a Django app. One instance runs on the cluster (though can be scaled up as any django app can). Through this API, you can deploy and interact with containers. The API delegates operations to either Fleet or the Satellites transparently, so the users of the API need not concern themselves with the internal operations, and simply see a single API.

Auth

The default user and password are set to admin/admin. To change, use the createpasswordhash.py to generate a password hash, and change the password field in data/data.json. If using containers, use the environment variable ADMINHASH.

How to deploy

  • Using the service files from Docker/units
  • fleetctl submit api.service satellite.service satellite-discovery.service proxy.service
  • fleetctl load api.service satellite.service satellite-discovery.service proxy.service
  • fleetctl start api.service satellite.service satellite-discovery.service proxy.service

HTTP Proxy

The API container also runs configurable-http-proxy to allow proxying of http from containers. It is not yet linked to the API though.

Data Proxy

The proxy container runs a squid proxy on all hosts, to allow for caching of frequently requested data.

Building Containers

In the Docker directory, run the build-and-push-docker.sh script. It takes two arguments, repo name, and tag, e.g bash ./build-and-push-docker.sh switcheu latest

Endpoints

  • GET /v1/container/ - List all running containers deployed via this API
  • POST /v1/container/ - Create a container. Script can either be in the json base64 encoded, or as a file in a multipart post named "script". Currently limited to 4KB. { "image_name":"ubuntu", "image_tag":"latest", "scripturl":"http://example.com/myscript.sh", "scriptname":"myscript.py", "dataurl":"http://example.com/mydata.zip (optional)", "datapath": "/path/inside/container (optional)", "container_args": "/extra/args --for=container (optional)", }
  • DELETE /v1/container/<container id>/ - Delete a container
  • GET /v1/container/<containerID>/ - Get Container Info
  • GET /v1/container/<containerID>/stdout - Get Container Output
  • GET /v1/container/<containerID>/script - Get script
  • GET /v1/container/<containerID>/state - Get state
  • GET /v1/container/<containerID>/metrics - Get Container Metrics
  • GET /v1/container/<containerID>/changes - Get a list of changed files in the Container
  • GET /v1/container/<containerID>/file - Get a list of changed files in the Container
  • GET /v1/container/<containerID>/file/<filepath> - Get a specific file from the Container
  • GET /v1/container/<containerID>/filebynum/<filenum> - Get a specific file from the Container (by position in array)
  • GET /v1/container/<containerID>/archive - Get full changes archive

State

State returns the state as provided by both fleet and docker in the form

{
    "fleet_state": {
        "name": "4b709b3c30df46bbad2c2280ff624dbc",
        "activeState": "loaded",
        "loadState": "loaded",
        "subState": "loaded"
    },
    "docker_state": {
        "Paused": false,
        "FinishedAt": "2016-05-13T16:01:57.091811098Z",
        "StartedAt": "2016-05-13T16:01:50.736134732Z",
        "Restarting": false,
        "Dead": false,
        "ExitCode": 0,
        "Error": "",
        "OOMKilled": false,
        "Pid": 0,
        "Running": false,
        "Status": "exited"
    }
}

docker_state will be blank if called before the container has been started. Note: there's a bug where an error is returned when called early, waiting and trying again works - needs fixing

stdout

This has changed slightly, so you may need to update any code speaking to it

Metrics

More metrics are returned now, and in a different format

Extra options

TODO

  • ☐ Implement extra options
  • ☑ Provision for ephemeral vs permanent containers (systemd restart=always etc)
  • ☐ Allow definition of ports to expose and store in etcd (ip:port)
  • ☐ Add ability to specify dict of environmental vars when creating containers, and emulate docker link, (grab ip+port from etcd, insert env var)
  • ☐ Allows templated env vars, e.g. #Containerid.port to get another containers port?
  • ☐ Eventually - Tosca?
  • ☐ Cleanup files on container deletion
  • ☐ Should we assume access to the URLs defined by scripturl and dataurl or copy them local (to Djangos file storage) and provide them from there?
  • ☐ Unzipping - make sure unzipping to correct place, not escaping directories
  • ☐ Sort out exceptions/errors to be more useful
  • ☐ Finish User/Security for the API
  • ☐ Finish implementing configurable-http-proxy for endpoint redirection using etcd. (Needed?)
  • ☐ Secure API to Satellite calls (TLS? HMAC? Both? Client Auth TLS?)
  • ☐ Expand API to allow more interaction with Fleet unit files and their capabilities
  • ☐ Decouple Fleet specific functionality into it's own module to allow different backends to plug in to a unified API.
  • ...