Convey

Convey is a pipeline for running containers and protecting the host as much as possible. It accomplishes this goal by creating a volume and making it available to each item in the pipeline.

Convey was created out of the need to build Pidgin and its supporting libraries on multiple Linux distributions, Windows and MacOS. While there are many continuous integration tools out there, all of them still require maintenance and can only be used by the project itself.

This becomes a problem when you have a community of plugin developers who would also like to build against all of its supported operating systems but can't and shouldn't need to put forth the effort and maintenance involved with using other available tools.

This is where Convey's concept of build containers comes into use. Once you have all the dependencies required to build something, like a Pidgin plugin, you are then able to share that Docker image with everyone using this system and they are then able to use that image to build against. So a project like Pidgin can supply build images containing the most recent releases with all of the build dependencies for multiple platforms and then plugin developers can use that image to test their plugins against.

This also works when it comes to deployments. You can create an image to deploy to bintray so that users are able to specify options in their convey.yml and not have to worry about the specifics of how it's uploaded.

If you still have doubts about how Convey can simplify your workflow and increase productivity, please contact me via email or by opening an issue and let me know!

License

GPLv3+

Concepts

Convey is built around a concept that I call a build container. A build container is a container that has all of the dependencies to turn an input into something else. This could be source code or even just data processing. For example the convey/go-build image is a simple wrapper around the golang image that just builds a project instead of installing it. You can find more information on build containers here.

Convey is built around this concept to allow you to build/process anything on any platform while at the same time protecting the host from a malicious config. This protection is provided by not exposing dangerous options like volume mounts or port forwards while running containers. This is accomplished by creating a volume a data volume at the start of the build plan and volume mounting it into containers as they are run. This allows you to pass state between modular tasks that can be reused between multiple plans.

Once built, convey can build itself entirely. This is done by using an import task to copy the source code into the workspace. Once that stage is complete, it runs another stage that builds for linux-x86_64, windows-x86_64, and darwin-x86_64 in parallel. The final stage will take those compiled binaries and export them back to the hosts file system.

Of course you don't have to export back to the host file system. Since these are just containers, you could instead upload them to your staging environment, packagecloud.io, bintray.com, whatever. Or you could export to the host and create another plan that will import the artifacts and then publish them.

Installation

There are a few ways you can install convey. The easiest is to grab it from the downloads section.

If you want to use the version currently in development and have a golang environment setup, you can run go install bitbucket.org/rw_grim/convey and it'll install into your $GOPATH.

Usage

usage: convey [<flags>] <command> [<args> ...]

Convey is a container pipeline runner.

Flags:
      --help                   Show context-sensitive help (also try --help-long and --help-man).
      --version                Show application version.
      --color                  Enable colorized output
  -l, --config-loader=convey   Select the configuration loader
  -f, --config=CONFIG          The config file name to use
  -c, --cpu-shares=CPU-SHARES  The amount of cpu shares to give to a run task
      --docker-config=DOCKER-CONFIG  
                               Location of docker client config files
  -e, --env=ENV ...            Set environment variables
  -S, --force-sequential       Don't run anything concurrently
  -m, --memory=MEMORY          The amount of memory to give the run task
      --ssh-agent              A shortcut for --ssh-identity=*
      --ssh-identity=SSH-IDENTITY ...  
                               Enable ssh-agent for the given identities
      --timeout=15m            The maximum amount of time a plan can run. 0 to disable. Units must be
                               specified.
  -v, --verbose                Be more verbose
      --disable-deprecated     Allow the user of deprecated features

Commands:
  help [<command>...]
    Show help.

  config
    Show a dump of the config file

  environment
    List the environment variables that are available

  graphviz
    Output a graphviz diagram of the config file

  run* [<plan>...]
    Run a plan or metaplan

  list environment
    List the environment variables that are available.

  list metaplans
    List the metaplans defined in the configuration file.

  list plans
    List the plans defined in the configuration file.

  list tasks
    List the tasks defined in the configuration file.

Configuration

Convey supports multiple configuration loader which can be specified by the -l command line argument.

Config Loaders

Convey supports multiple different types of configs that it can load. Currently supported loaders are convey and bitbucket.

Convey

Documentation for the built in format can be found in REFERENCE.md.
The convey config is the one described above and is the default.

Bitbucket

The bitbucket config is for bitbucket-pipelines which will allow you to run your pipeline locally. Convey does a best attempt at emulating the pipelines environment, but your mileage may vary. If you're having problems, please open an issue.

To use the bitbucket config loader, just run convey -l bitbucket. If there is a bitbucket-pipelines.yml file in the working directory, convey will happily run it.

bitbucket pipelines does not have plans in the same sense that convey does, so if you do not specify one, convey will use the same heuristics that bitbucket-pipelines does to run the correct pipeline from the config.

Also, since the bitbucket-pipelines.yml file is converted into the convey config format, --graphviz, --list-plans, and --list-tasks all work like normal as well.