Overview

The Stream-Of-Consciousness Guide to Minuteman Copyright 2011 by Larry Hastings

written at/for PyCon 2011 March 12th, 2011

First: if you're reading this, I'm sorry. The documentation is terrible. I used to write documentation in a Tiddlywiki; you can find it in "mm/readme.html". But that document is toxically out of date. At best you should use it as a springboard to discover new things about Minuteman. But please do not rely on it. My eventual goal is to replace that (and this) with some sane Sphinx-generated documentation. Watch this space.

Second: be sure to try the "rigged demo" that I used on stage at PyCon 2011. Download

http://larryhastings.com/minuteman/releases/mm.demo.0.8.zip

and follow the instructions in there.

Third: the rest of this document is basically a stream-of-consciousness dumping ground for Minuteman documentation topics. I'm sure it is dreadfully incomplete, and bafflingly organized, but it's current and it's your only chance. For now, that is.

Fourth: if this file is a little woozy, well, I am too. I was hit with the quintuple-whammy of

  • noisy neighbors up past 1am
  • noisy emergency vehicles cruising past at all hours
  • losing an hour of sleep to Daylight Savings
  • getting up extra-early forbreakfast and the morning lightning talks
  • being so tired I couldn't sleep

So I'm not running on much sleep. Believe me, I'm going to bed early tonight-- and getting up late.

Fifth: remember,
mm help
prints out all commands, and
mm help command

would print out long help on "command".

Onward To Glory!

How would you describe what a hammer is? If you said "it's what you use to pound nails into the wall", I might point out that that's a description of what a hammer is used for, not what it is. A hammer is a heavy weight with a flat part mounted on a handle. Pedantry aside, my point is, there's a big conceptual difference between what Minuteman is and what it's used for.

Minuteman is a program and set of libraries that makes it easy to create a run-time collection of objects ("projects") with a consistent external interface. This interface allows Minuteman to interrogate the object, and to instruct the object to perform actions.

With that, and with a lot of sensible defaults and the aforementioned libraries, Minuteman makes a handy generic large-software-system builder.

Minuteman Concepts And Taxonomy

A "workspace" is a self-contained directory tree. In a workspace you'll find "src", which is where all the projects go, and "release", which is where all the built software goes.

I use the term "project" really to mean two things: primarily, a run-time object conforming to the mm.Project object interface, but secondarily a directory on disk (in your workspace under "src/") that will be represented at runtime by the aforementioned run-time object.

An "action" is an object representing some verb--build, clean, document, test, regress--that Minuteman may request a project to perform. Minuteman can tell a project to "build", or "test", or whatnot.

A "step" is a single step in "building" a project--or whatever requested action. It's intended to map to a single executed external program, like an invocation of "configure". However, for convenience's sakes, commonly-used idioms like "make && make install" or "setup.py build && setup.py install" are bundled together into a single "step" for your convenience--though in actuality those are "composite steps", which run multiple internal steps for you. Currently defined steps that you might want to call directly:

preconfigure configure make make_install setup_py setup_py_install

Minuteman Code Of Ethics

Whenever you build, you install. Installation is always local, into "release". Inside "release" are "bin", "lib", "share", and so on--it's like the "--prefix" directory passed in to a "configure" script. (In fact it's exactly like that!)

All builds and installs are local. Nothing should change outside the workspace when you run a build.

The Demo

Here I'll walk you through what the demo does, in order.

% mm init ws1

This creates a new empty "workspace".

% mm addfetcher myhg hg ~/hg

This adds a "fetcher". Fetchers are objects that go and get source code for you. This fetcher is named "myhg", it is of type "hg" (Mercurial), and the URL it should use is "~/hg". After this, when you ask Minuteman to "add" a project, it'll look to see if there's a valid Mercurial repository in "~/hg/{name-of-project}".

% mm add libevent-python

This tells Minuteman "please add a project named libevent-python to the workspace". If you already have a "src/libevent-python", Minuteman will load whatever it finds in it; if you don't, Minuteman will try the fetchers to see if any of them can get one.

% mm

This runs a build. You can tell Minuteman to only build certain projects:
% mm libevent
You can tell Minuteman to clean the workspace:
% mm clean
You can tell Minuteman to clean only specific projects:
% mm clean libevent

% mm tag ../tagfile

This creates a "tagfile". Really all this does is copy Minuteman's workspace configuration to another file, after checking that no projects have outstanding changes. For a good time, take a look at the two files in Minuteman's secret directory. Go into ".mm" in your workspace and look at "configuration" and "local_configuration".

% mm clone tagfile ws2

This "clones" from the tagfile, creating a new workspace and populating it with the projects enumerated in the tagfile. If it could load all of the projects, it overwrites the workspace's configuration with the tagfile (to preserve all the settings) and pronounces success.

Making Your Own Projects With Minuteman

To experiment with having Minuteman build your own projects, you'll need: * a fresh workspace * a copy of your source (you don't have to check it in)

Let's create a new workspace to build a hypothetical project named "spacegoblin". First, create a new workspace using "mm init". Second, make a directory in the workspace at "src/spacegoblin", and copy your source files in there. Third, create a file called "mmproject.py" in that directory that looks like this (remove the first level of indent):

import mm

class Project(mm.Project):
def build(self):
# steps go here

Where I have the comment "steps go here", add one of two sets of things: * If your project is built with "configure && make && make install", add

self.configure() self.make_install()
  • If your project is built with "python setup.py build install", add

    self.setup_py_install()

    By default this runs Python 2; to use it with Python 3, change it to

    self.setup_py_install(python="python3")

    Here the string value is the path to the Python interpreter you want to use; without an absolute/relative path, Minuteman will look along the $PATH.

If you want to add a second project, and have one depend on the other, add a "requirements = " line at class scope:

class Project(mm.Project):

requirements = ["other-project-name"] def build(self):

# steps go here

"requirements" means projects that must be present, and will be built before the current project. "weak_requirements" means projects that aren't necessary, but if they are present in the workspace will be built before the current project.

Plugins: Loaders

The "loader" is easily the most powerful plugin in Minuteman.

Once first surprising thing about the project object--projects don't know their own name, implicitly. Rather, Minuteman tells the project what its name is, when it's loaded.

This is wholly deliberate. I think of this as "slip" in the interface, like the clutch on a car. It's a point of interface between two systems where I've built in some deliberate flexibility.

The "loader" interface looks like this:
def load(prototype) -> project_object

The "prototype" is a prototype of the project--it's a generic object set up to kinda look like a project, but it isn't really a project. The project prototype is required to have a couple of important bits of metadata:

name settings dict directory (if one exists)
The settings dict is straight out of the configuration file,
configuration["projects][name]["settings"]

Minuteman has four builtin loaders of interest:

  • ImportLoader

The ImportLoader is the loader that looks in the project's directory for a "mmproject.py" file. If the loader finds one, it imports it by hand. If the import works, the loader looks to see if it has a "Project". If it has one, the loader attempts to call it as if it were a Project constructor. If that returns non-None, the loader returns that.

  • ProxyLoader

The ProxyLoader lets you load a project where the project and its mmproject.py come from different directories. This is for when you use a repository where the projects don't have "mmproject.py" files of their own, but you have to have an explicit mmproject.py. Just create a project with a directory, where the load name of the class is "mm ProxyLoader", then place in the directory files named with the name of the project you want to load. For example, to use ProxyLoader with libevent, you'd have a "libevent.py" in the ProxyLoader project directory. If you tried to load "libevent" with the ProxyLoader, it's load that "libevent.py", pointed at the "libevent" source directory. (This is one reason why the "load name" is a useful bit of slippage--you can rename the project in your workspace without losing the ability to point a ProxyLoader at the correct Python script.)

  • InferredLoader

If you don't have any sort of Python code to handle a project, the InferredLoader may be able to help. It looks in a directory to see if there are any standard idiomatic build scripts it can recognize; if it sees one, it creates a default project object understanding that build and returns it. For example, if it sees a "configure" script, it returns a generic project whose "build" runs the "configure" step then the "make_install" step.

  • VirtualLoader

VirtualLoader loads projects without having to hit the disk--they should already be loaded into the interpreter. The way it works is, you give the VirtualLoader a dict to scan over of preknown project names mapping to project class objects. You can also specify a required prefix for the project name. "mm trace" is loaded by a VirtualLoader, as are all the built in loaders and fetchers.

Plugins: Fetchers

A Fetcher is a plugin that goes and fetches source code for you. When you "add" a project, it's supposed to go something like this:

Try and load it by name. If that worked, return it. For each fetcher

Try and fetch it by name. If that succeeded:

Try and load it by name. If that worked, return it. Remove the directory.

A fetcher knows what "type" it is ("hg", "git", or "svn"), and what URL to go get the source from. The URL uses modern str.format string substitution, allowing the following fields:

{name} - the name of the project to fetch {revision} - the revision of the project {branch} - what branch to get the project from (an argument to "add")

A"fetcher" exposes a lot of interfaces, but the main one is "fetch". It looks a lot like a loader's "load" method, to whit:

def fetch(prototype) -> bool (success/failure)

One clever fetcher method: fetcher.detect() attempts to determine "could this directory have been checked out from me"? It does this by figuring out the URL used to check out the directory, then turns its URL into a regular expression (where "{name}" becomes something like "(?P<name>)"). If the regular expression matches, the groups of the match object tell us the parameters to the original "fetch" request.

Plugins: Mutate

A mutator is a plugin that monitors actions and steps as they are executed. There's one sample mutator; you can add it to your workspace with

% mm add "mm trace"

The "trace" mutator

"name", "load name", and "fetch name"

Here's one example of flexibility allowed by the "slip" of projects not knowing their own names. When you load it, you can use different names at every step.

"name" is the name the Minuteman workspace uses for the project. You look up a project in workspace.projects[name] with this name.

"load name" is the name the loader uses when loading the project. If unspecified, it defaults to "name".

"fetch name" is the name the fetcher uses when fetching the project. It's the {name} substituted in to the fetcher URI. If unspecified, it defaults to "load name" (and therefore in turn to "name").

BoundInnerClass

I use my BoundInnerClass class decorator a lot. You can read about it here, and thankfully it has good documentation:

http://code.activestate.com/recipes/577070-bound-inner-classes/

commandline.CommandLine

This class is basically terrible. Maybe the best thing is to require Python 3.2 and switch to argparse, or use something like commandline.CommandLine for the initial work and use argparse the rest of the way, or something.