mm /

Filename Size Date modified Message
doc
mm
mm_loader
130 B
55 B
844 B
1014 B
12.9 KB
1.9 KB
The Stream-Of-Consciousness Guide to Minuteman
Copyright 2011 by Larry Hastings

written at/for PyCon 2011
March 12th, 2011


First: if you're reading this, I'm sorry.  The documentation is terrible.
I used to write documentation in a Tiddlywiki; you can find it in
"mm/readme.html".  But that document is *toxically* out of date.  At best you
should use it as a springboard to discover new things about Minuteman.  But
please do *not* rely on it.  My eventual goal is to replace that (and this)
with some sane Sphinx-generated documentation.  Watch this space.


Second: be sure to try the "rigged demo" that I used on stage at PyCon 2011.
Download
    http://larryhastings.com/minuteman/releases/mm.demo.0.8.zip
and follow the instructions in there.

Third: the rest of this document is basically a stream-of-consciousness dumping
ground for Minuteman documentation topics.  I'm sure it is dreadfully
incomplete, and bafflingly organized, but it's current and it's your only
chance.  For now, that is.

Fourth: if this file is a little woozy, well, I am too.  I was hit with the
quintuple-whammy of
    * noisy neighbors up past 1am
    * noisy emergency vehicles cruising past at all hours
    * losing an hour of sleep to Daylight Savings
    * getting up extra-early forbreakfast *and* the morning lightning talks
    * being so tired I couldn't sleep
So I'm not running on much sleep.  Believe me, I'm going to bed early tonight--
and getting up late.

Fifth: remember,
    mm help
prints out all commands, and
    mm help command
would print out long help on "command".


Onward To Glory!

How would you describe what a hammer is?  If you said "it's what you use to
pound nails into the wall", I might point out that that's a description of
what a hammer is *used for*, not what it *is*.  A hammer *is* a heavy weight
with a flat part mounted on a handle.  Pedantry aside, my point is, there's a
big conceptual difference between what Minuteman is and what it's used for.

Minuteman *is* a program and set of libraries that makes it easy to create
a run-time collection of objects ("projects") with a consistent external
interface.  This interface allows Minuteman to interrogate the object, and to
instruct the object to perform actions.

With that, and with a lot of sensible defaults and the aforementioned libraries,
Minuteman makes a handy generic large-software-system builder.


Minuteman Concepts And Taxonomy

A "workspace" is a self-contained directory tree.  In a workspace you'll find
"src", which is where all the projects go, and "release", which is where all
the built software goes.

I use the term "project" really to mean two things: primarily, a run-time object
conforming to the mm.Project object interface, but secondarily a directory on
disk (in your workspace under "src/") that will be represented at runtime by
the aforementioned run-time object.

An "action" is an object representing some verb--build, clean, document, test,
regress--that Minuteman may request a project to perform.  Minuteman can tell a
project to "build", or "test", or whatnot.

A "step" is a single step in "building" a project--or whatever requested action.
It's intended to map to a single executed external program, like an invocation
of "configure".  However, for convenience's sakes, commonly-used idioms like
"make && make install" or "setup.py build && setup.py install" are bundled
together into a single "step" for your convenience--though in actuality those
are "composite steps", which run multiple internal steps for you.  Currently
defined steps that you might want to call directly:
    preconfigure
    configure
    make
    make_install
    setup_py
    setup_py_install


Minuteman Code Of Ethics

Whenever you build, you install.  Installation is always local, into "release".
Inside "release" are "bin", "lib", "share", and so on--it's like the "--prefix"
directory passed in to a "configure" script.  (In fact it's *exactly* like
that!)

All builds and installs are local.  Nothing should change outside the workspace
when you run a build.


The Demo

Here I'll walk you through what the demo does, in order.

% mm init ws1

This creates a new empty "workspace".

% mm addfetcher myhg hg ~/hg

This adds a "fetcher".  Fetchers are objects that go and get source code for
you.  This fetcher is named "myhg", it is of type "hg" (Mercurial), and the
URL it should use is "~/hg".  After this, when you ask Minuteman to "add" a
project, it'll look to see if there's a valid Mercurial repository in
"~/hg/{name-of-project}".

% mm add libevent-python

This tells Minuteman "please add a project named libevent-python to the
workspace".  If you already have a "src/libevent-python", Minuteman will
load whatever it finds in it; if you don't, Minuteman will try the fetchers
to see if any of them can get one.

% mm

This runs a build.  You can tell Minuteman to only build certain projects:
    % mm libevent
You can tell Minuteman to clean the workspace:
    % mm clean
You can tell Minuteman to clean only specific projects:
    % mm clean libevent

% mm tag ../tagfile

This creates a "tagfile".  Really all this does is copy Minuteman's workspace
configuration to another file, after checking that no projects have outstanding
changes.  For a good time, take a look at the two files in Minuteman's secret
directory.  Go into ".mm" in your workspace and look at "configuration" and
"local_configuration".

% mm clone tagfile ws2

This "clones" from the tagfile, creating a new workspace and populating it with
the projects enumerated in the tagfile.  If it could load all of the projects,
it overwrites the workspace's configuration with the tagfile (to preserve all
the settings) and pronounces success.


Making Your Own Projects With Minuteman

To experiment with having Minuteman build your own projects, you'll need:
* a fresh workspace
* a copy of your source (you don't have to check it in)

Let's create a new workspace to build a hypothetical project named
"spacegoblin".  First, create a new workspace using "mm init".  Second, make
a directory in the workspace at "src/spacegoblin", and copy your source files
in there.  Third, create a file called "mmproject.py" in that directory that
looks like this (remove the first level of indent):
    import mm

    class Project(mm.Project):
        def build(self):
            # steps go here

Where I have the comment "steps go here", add one of two sets of things:
* If your project is built with "configure && make && make install", add
            self.configure()
            self.make_install()
* If your project is built with "python setup.py build install", add
            self.setup_py_install()
  By default this runs Python 2; to use it with Python 3, change it to
            self.setup_py_install(python="python3")
  Here the string value is the path to the Python interpreter you want to
  use; without an absolute/relative path, Minuteman will look along the $PATH.


If you want to add a second project, and have one depend on the other, add
a "requirements = " line at class scope:

    class Project(mm.Project):
        requirements = ["other-project-name"]
        def build(self):
            # steps go here

"requirements" means projects that *must* be present, and will be built before
the current project.
"weak_requirements" means projects that aren't necessary, but if they are
present in the workspace will be built before the current project.


Plugins: Loaders

The "loader" is easily the most powerful plugin in Minuteman.

Once first surprising thing about the project object--projects don't know
their own name, implicitly.  Rather, Minuteman *tells* the project what its
name is, when it's loaded.

This is wholly deliberate.  I think of this as "slip" in the interface, like
the clutch on a car.  It's a point of interface between two systems where
I've built in some deliberate flexibility.

The "loader" interface looks like this:
    def load(prototype) -> project_object
The "prototype" is a prototype of the project--it's a generic object set up to
kinda look like a project, but it isn't really a project.  The project prototype
is required to have a couple of important bits of metadata:
    name
    settings dict
    directory (if one exists)
The settings dict is straight out of the configuration file,
    configuration["projects][name]["settings"]


Minuteman has four builtin loaders of interest:

* ImportLoader
The ImportLoader is the loader that looks in the project's directory for
a "mmproject.py" file.  If the loader finds one, it imports it by hand.  If the
import works, the loader looks to see if it has a "Project".  If it has one,
the loader attempts to call it as if it were a Project constructor.  If that
returns non-None, the loader returns that.

* ProxyLoader
The ProxyLoader lets you load a project where the project and its mmproject.py
come from different directories.  This is for when you use a repository where
the projects don't have "mmproject.py" files of their own, but you have to
have an explicit mmproject.py.  Just create a project with a directory, where
the load name of the class is "mm ProxyLoader", then place in the directory
files named with the name of the project you want to load.  For example, to
use ProxyLoader with libevent, you'd have a "libevent.py" in the ProxyLoader
project directory.  If you tried to load "libevent" with the ProxyLoader, it's
load that "libevent.py", pointed at the "libevent" source directory.  (This is
one reason why the "load name" is a useful bit of slippage--you can rename the
project in your workspace without losing the ability to point a ProxyLoader at
the correct Python script.)

* InferredLoader
If you don't have any sort of Python code to handle a project, the
InferredLoader may be able to help.  It looks in a directory to see if there
are any standard idiomatic build scripts it can recognize; if it sees one, it
creates a default project object understanding that build and returns it.
For example, if it sees a "configure" script, it returns a generic project whose
"build" runs the "configure" step then the "make_install" step.

* VirtualLoader
VirtualLoader loads projects without having to hit the disk--they should already
be loaded into the interpreter.  The way it works is, you give the VirtualLoader
a dict to scan over of preknown project names mapping to project class objects.
You can also specify a required prefix for the project name.  "mm trace" is
loaded by a VirtualLoader, as are all the built in loaders and fetchers.


Plugins: Fetchers

A Fetcher is a plugin that goes and fetches source code for you.  When you
"add" a project, it's supposed to go something like this:

    Try and load it by name.
    If that worked, return it.
    For each fetcher
        Try and fetch it by name.
        If that succeeded:
            Try and load it by name.
            If that worked, return it.
            Remove the directory.

A fetcher knows what "type" it is ("hg", "git", or "svn"), and what URL to
go get the source from.  The URL uses modern str.format string substitution,
allowing the following fields:
    {name} - the name of the project to fetch
    {revision} - the revision of the project
    {branch} - what branch to get the project from (an argument to "add")

A"fetcher" exposes a lot of interfaces, but the main one is "fetch".  It looks
a lot like a loader's "load" method, to whit:
    def fetch(prototype) -> bool (success/failure)

One clever fetcher method: fetcher.detect() attempts to determine "could this
directory have been checked out from me"?  It does this by figuring out the URL
used to check out the directory, then turns its URL into a regular expression
(where "{name}" becomes something like "(?P<name>)").  If the regular expression
matches, the groups of the match object tell us the parameters to the original
"fetch" request.


Plugins: Mutate

A mutator is a plugin that monitors actions and steps as they are executed.
There's one sample mutator; you can add it to your workspace with
    % mm add "mm trace"
The "trace" mutator


"name", "load name", and "fetch name"

Here's one example of flexibility allowed by the "slip" of projects not
knowing their own names.  When you load it, you can use different names at
every step.

"name" is the name the Minuteman workspace uses for the project.
You look up a project in workspace.projects[name] with this name.

"load name" is the name the loader uses when loading the project.
If unspecified, it defaults to "name".

"fetch name" is the name the fetcher uses when fetching the project.
It's the {name} substituted in to the fetcher URI.  If unspecified,
it defaults to "load name" (and therefore in turn to "name").


================
Technology Notes

BoundInnerClass

I use my BoundInnerClass class decorator a lot.  You can read about it here,
and thankfully *it* has good documentation:
    http://code.activestate.com/recipes/577070-bound-inner-classes/

commandline.CommandLine

This class is basically terrible.  Maybe the best thing is to require Python
3.2 and switch to argparse, or use something like commandline.CommandLine for
the initial work and use argparse the rest of the way, or something.
Tip: Filter by directory path e.g. /media app.js to search for public/media/app.js.
Tip: Use camelCasing e.g. ProjME to search for ProjectModifiedEvent.java.
Tip: Filter by extension type e.g. /repo .js to search for all .js files in the /repo directory.
Tip: Separate your search with spaces e.g. /ssh pom.xml to search for src/ssh/pom.xml.
Tip: Use ↑ and ↓ arrow keys to navigate and return to view the file.
Tip: You can also navigate files with Ctrl+j (next) and Ctrl+k (previous) and view the file with Ctrl+o.
Tip: You can also navigate files with Alt+j (next) and Alt+k (previous) and view the file with Alt+o.