HTTPS SSH

About

The goal of this project is to aggregate and expose repository and automation data for Mozilla projects and to facilitate richer integration through that data.

Achitecture

This project consists of a number of different components working in harmony. Read the sections below for details on each component and its pieces.

repodata

The repodata project/Python package aims to aggregate and expose repository data. It does this by locally mirroring official Mozilla repos (this is needed in order to access the vast amounts of data in the repos) and then extracting details from those repositories into a database. In some cases, repo data not part of clones repositories (such as pushlog data) is fetched from Mozilla servers and inserted into a database. Finally, the repodata project provides an HTTP server that exposes aggregated data.

Background Service

A background service is in charge of coordinating all the work and processes required to support the project. The background service is managed by circus.

Read below for details on how to manage the service.

Mercurial Web Server

The project exposes a Mercurial web server (hgweb) that serves Mercurial repositories and allows HTML clients to view repository data.

Configuration and Running

To run this project, you'll need to set up an environment.

The easiest way to do this is to install Vagrant and use the included Vagrantfile to automatically configure a virtual machine:

$ vagrant up

If provisioning fails for whatever reason, try again:

$ vagrant provision

Please note that provisioning takes a few minutes to complete.

The Vagrant virtual machine will be attached to 10.0.100.100 on a private network. It will expose a Postgres server on port 5432. There exists a mozilla role with password mozilla. It has permissions on the moztree database.

You'll want to do everything as the mozilla user in the VM:

$ vagrant ssh
$ sudo su - mozilla

A Python virtualenv is created as part of Vagrant provisioning. It's a good idea to source it into the current shell so all the commands are available:

$ source ~/env/bin/activate

Config Files and Environment Variables

A config file is necessary to run pretty much anything in this project. The default config.ini file in the source tree is documented and should give you an idea of what all can be configured.

It is common practice to set the REPODATA_CONFIG_FILE environment variable to the path of the active config file. Alternatively, the config file location may be specified in arguments to invoked commands.

Running the Daemon

To run the background daemon, run:

$ moztree-daemon run /path/to/config/file

The process will sit in the foreground, so you may want to run it inside screen.

Database Migrations

The project integrates Alembic to manage database migrations. Run alembic like so:

$ alembic --config config.ini upgrade tip