smart-live-rebuild / README

smart-live-rebuild
(C) 2010 Michał Górny <gentoo@mgorny.alt.pl>
Released under the terms of the 3-clause BSD license or the GPL-2 license.


smart-live-rebuild is a Python script to aggregate live packages from users'
system, update them and merge the updated ones.


MOTIVATION
-----------
What's the point in creating such tool when portage-2.2 comes with builtin
@live-rebuild support?

The @live-rebuild set of portage and similar tools simply do the task of aggre-
gating the live packages and calling emerge to update them. This is acceptable
if you don't have many live packages and don't call it frequently.

But if you used a whole bunch of live packages (like live X11, for example),
you'd probably get a little irritated having to rebuild all of them, even if
most of them weren't even touched by upstream.

Of course, there's LIVE_FAIL_blah_blah in git.eclass. It works indeed but have
you used it with, say, more than 90 packages? You can imagine how long it takes
for portage to reload itself 90 times after dying on each non-modified repo.

And that's where smart-live-rebuild comes in handy. It updates all the live
packages on its own, and supplies portage with ready-made list of packages which
indeed have changed and need to be rebuilt.


FEATURES
---------
1) parallel updates support

As you might have already noticed, fetching from remote repositories suffers
from a constant start-lag, wasting a lot of time without really consuming
network bandwidth.

To overcome that, smart-live-rebuild support running multiple updates in para-
llel. Although it might seem insane at first, assuming that most repositories
aren't really changed or have quite a small set of changes, using it could save
a lot of time.

To use parallel updates, pass the '-j N' ('--jobs N') parameter to s-l-r, where
N stands for the number of updates running in parallel.

2) quickpkg backup support

If you used a large set of live packages for some time, then you probably met
an issue where the new upstream version was broken, and you had to make
an effort in order to find a working commit to revert to.

This is where the '-Q' ('--quickpkg') option becomes useful. It makes s-l-r call
'quickpkg' to create the binary packages for the current versions of packages
queued to be updated.

Although it wastes some time in each update, it allows you to easily and quickly
revert to the previous working version of the package -- without unnecessarily
recompiling it. Moreover, the binary packages contain a copy of environment.bz2,
allowing you to get the 'last surely working commit'.

3) real update checking

s-l-r doesn't assume it is the only application which can update the repository.
Whenever possible, it tries to retrieve the currently-installed commit (revi-
sion) from vardb instead of relying on the repository state before update.

This way, the rebuild list will contain all packages which are outdated
in the system and not only those which it has updated in the particular run.
This means you no longer have to worry about having to manually update packages
whenever you abort the merge or update process.


SECURITY
---------
In most cases, s-l-r needs to be run as root. This has serious security impli-
cations, which the scripts tries to overcome dropping the privileges whenever
possible.

In fact, the superuser privileges are only required to call emerge and quickpkg
(the latter might in some cases work as regular user but that has further
implications). It is also required to update the repositories unless userpriv is
enabled (otherwise, portage privileges are enough).

s-l-r loses its privileges as soon as configuration and command-line options are
parsed (this is required in order to support disabling the privilege dropping),
and performs the repository updates as the 'portage' user (as long as userpriv
is enabled).

If 'quickpkg' is scheduled and/or '--pretend' is not being used, s-l-r forks to
drop the privileges and performs the updates using forked subprocess. Otherwise,
it directly drops the privileges in the parent process.

Moreover, in the latter case s-l-r can be run directly by the 'portage' user.


PORTAGE SET SUPPORT
--------------------
Apart from being called directly, smart-live-rebuild might also be used through
portage-2.2 CommandOutputSet. In order to do so, you need to add it to your
sets.conf (/etc/portage/sets.conf) file. You may see the 'sets.conf.example'
file for reference.

Please notice that if you don't have the 'psutil' Python module (required to
traverse the process tree) you need to explicitly specify '--pretend' ('-p')
when being called from sets.conf. Otherwise, the program will spawn second
emerge instance.


CONFIGURATION FILE
-------------------
Various options to smart-live-rebuild may be set in a configuration file too.
The path to that file can be specified as '--config-file' (or '-c'), and
the default one is '/etc/portage/smart-live-rebuild.conf'.

The configuration file has a format similar to Windows .ini files. Sections
correspond to available profiles, with '[smart-live-rebuild]' being the default
one (other can be specified using '--profile'/'-p'). The keys for respective
options match the command-line argument names, with 'no-' prefix stripped
and all dashes ('-') replaced with underscores ('_').

The values for boolean types can be specified in any way suitable for
ConfigParser python class (i.e. true/false/yes/no/on/off). The 'type' list
has to be comma-separated.

Example config file may look like:

[smart-live-rebuild]
jobs=3
erraneous_merge=no
type=git,subversion

Additionally, configuration files can be chained. To do so, just specify path
to the next configuration file as 'config_file'. Please notice though that
currently they aren't read in reverse order, so further configuration files will
replace values set by yours.


CONTACT / BUG REPORTING
------------------------
http://mgorny.alt.pl/contact.html
Tip: Filter by directory path e.g. /media app.js to search for public/media/app.js.
Tip: Use camelCasing e.g. ProjME to search for ProjectModifiedEvent.java.
Tip: Filter by extension type e.g. /repo .js to search for all .js files in the /repo directory.
Tip: Separate your search with spaces e.g. /ssh pom.xml to search for src/ssh/pom.xml.
Tip: Use ↑ and ↓ arrow keys to navigate and return to view the file.
Tip: You can also navigate files with Ctrl+j (next) and Ctrl+k (previous) and view the file with Ctrl+o.
Tip: You can also navigate files with Alt+j (next) and Alt+k (previous) and view the file with Alt+o.