1. Michael Forbes
  2. gobo_install

Overview

HTTPS SSH

README for gobo_install

This package simplifies the process of getting a GoboRootless installation up and running. It uses configurations specified in the Installs directory to define and build custom compile environments.

Overview

Gobo Rootless allows you to install a "mock" Gobo Linux distribution without root access. Everything will be installed in a specified directory that you can use by running a shell script to setup your environment. Many useful tools are provided for compiling custom software, managing dependencies, etc. Particularly useful is the Recipe mechanism for quickly specifying dependencies and how to compile new software.

This package simplifies the installation of the Gobo Rootless environment on a host system, allowing you to use previously installed software (such as compilers, libraries etc.)

Disclaimer

The gobo_install software is still in beta testing stage.

Usage

  1. Download and unpack this package into a directory (gobo_install in the example below.)
  2. Make a directory Installs/new_env (choose whatever name you want). Below we copy the Installs/template directory as a starting example.
  3. Modify the following files in Installs/new_env as needed:
    1. configuration.sh: A configuration file.
    2. Recipes: Add any customized recipes you need.
    3. HostPrograms: A list of the programs provided by the host system.
  4. Run the bin/goborootless.sh command which will build the installation.
  5. Source the file Install/<env_name>/build/environment.bash to start using the new installation.

Now you can try installing stuff from previously existing Gobo recipes, or you can edit or make your own recipes.

Example

Here is a complete example, copying the required files from Installs/template:

hg clone https://bitbucket.org/mforbes/gobo_install gobo_install
cd gobo_install
hg sum
cd Installs/
mkdir new_env
cp -r template/* new_env/
cd new_env/
../../bin/goboinstall.sh -h   # Display usage message
../../bin/goboinstall.sh
bash --init-file "build/environment.bash"
MakeRecipe http://www.python.org/ftp/python/2.7/Python-2.7.tar.bz2
n

Compile Python 2.7

If you have all of the required software for the install, this takes about a minute or so. On select systems (Mac OS X and some versions of Linux), if you do not have the required software, then a native package management system (MacPorts in the OS X case) will be used to install these. (There are some command line options to change this behaviour.)

See below for a detailed play-by-play.

Recipes

To install new software, you can first try to install it from an existing recipe using the Compile command:

Compile Python 2.7

If a suitable recipe does not exist, then you can edit/update an existing recipe or create a new one from a source distribution:

NewVersion :
NewVersion Python 2.7
EditRecipe :
EditRecipe Python
MakeRecipe :
MakeRecipe http://www.python.org/ftp/python/2.7/Python-2.7.tar.bz2

These will install the new recipe in Install/new_env/Recipe where you can inspect and customize it as required. For more details see:

I also maintain a set of recipes that I have edited and developed while using this project in my gobo_recipes project and I have a super-repository gobo that includes both gobo_install and gobo_recipes:

Dependencies

The original intent of Gobo Linux was to provide a complete linux distribution. As such, it generally tries to install all of the required dependencies. For a rootless installation this is undesirable. Instead, we try to make use of existing programs.

Note that some of the programs are required to support GNU specific conventions. On the Mac OS X platform this is especially a problem. The install script attempts to locate these programs using the paths specified in Installs/new_env/configuration.sh. Note that are several paths here including some that specify tools only needed by the install scripts, and other specifying programs that should be included as part of the final environment.

One option for dealing with dependencies is to link the appropriate programs to the Installs/new_env/build/sbin directory. This allows you to clean the path, thereby better controlling the dependencies in the environment.

The install scripts will check for the required programs, emitting errors if they do not exist: you will have to deal with these manually. The following prerequisites are noteworthy (they caused problems when trying to install on Scientific Linux):

  • python >= 2.3: The original Gobo Scripts package, although stating that it only requires 2.3, uses features requiring up to 2.5. I have included a patch that removes this requirement from the scripts typically used for a Rootless install. Your first step after installing, however, should be to update python to a reasonable version, for example:

    Compile Python 2.7
    
  • GCC != 4.1.x: This compiler has very poor floating point performance, so in some of my Recipes, I have excluded it. You may need to install a new version:

    Compile GCC
    

HostPrograms

If your host operating system provides programs that you wish to use in the Gobo setting, then you should specify this in the file specified by the hostPrograms variable (defaults to Installs/new_env/HostPrograms). This should contain a list of programs (using the Gobo names as listed in the Gobo recipes database) that need not be installed because the exist on the system outside of the current installation. The format is a series of names, or name version pairs:

GCC 4.1.2
Bash
Python 2.7

When a version is specified, a directory in ${goboPrefix}/Programs such as ${goboPrefix}/Programs/GCC/4.1.2 will be created (with only a file ProvidedByHost). When a version is not specified, the program is added to ${goboPrefix}/Programs/Scripts/Settings/Scripts/Dependencies.blacklist. These will be used by the CheckDependencies program to determine if they need to be compiled or installed.

In a future version of the gobo_install package, an Autoconf configure script will be provided to try and populate the HostPrograms file as accurately as possible.

Further Information

Further information about this particular set of install scripts can be found at the project homepage:

http://bitbucket.org/mforbes/gobo_install

Details about the general Gobo Linux distribution can be found here:

http://www.gobolinux.org/

A list of available recipes can be found here:

http://recipes.gobolinux.org/r/

Example Details

Here is a play by play of the example given earlier. First we checkout the gobo_install package and check the version downloaded:

$ hg clone https://bitbucket.org/mforbes/gobo_install gobo_install
   requesting all changes
   ...
   updating to branch default
   99 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ cd gobo_install
$ hg sum
   parent: 27:9ab928efc98e tip
   ...

Now we make a new directory in the Installs folder for our new environment which we call new_env. We copy the template files into this as a starting point:

$ cd Installs
$ mkdir new_env
$ cp -r template/* new_env/

Now we go into this directory and run the goboinstall.sh script. This takes about a minute and downloads a few essential components, as well as sets up the environmental files, locates some locally installed programs etc.:

$ cd new_env/
$ ../../bin/goboinstall.sh -h   # Display usage message

  Usage: goborootless.sh [OPTION]
  Setup a gobo rootless installation for building custom software.  This
  will create a directory <install folder> in which all applications
  will be built.  It will also setup initialization scripts for setting
  up your environment to use the software in that folder.

  Options:
    -h                  Display this help message.
    -q                  Query the user before performing actions.
    -n                  Don't actually run any commands; just print them.
    -v                  Verbose.
    -s                  Run MacPorts or aptitude using sudo.
    -c <configFolder>   Get configuration from this folder (by default
                        this is the current directory).
    -i <goboPrefix>     Create the instalation in this directory.
                        Default is in a folder 'build' in the
                        <configFolder>.
    -a <archive_folder> Tarballs will be downloaded here.  If this is
                        specified, then several installations can share
                        these preventing the need for multiple
                        downloads.  If provided, then
                        /Files/Compile/Archives will be
                        linked here.
    -Q                  Quarantine mode: the environment is completely
                        reset before building: only installed programs
                        or those symlinked to the sbin directory will be
                        available.

$ ../../bin/goboinstall.sh
   You are about to create a Rootless environment at
   .../gobo_install/Installs/new_env/build
   Creating ...
   ...
   To start the new_env environment start a shell as follows:
     bash --init-file ".../gobo_install/Installs/new_en

Now we can start a new shell with the Gobo environment. This is how you should use the environment in the future as well.:

$ bash --init-file "build/environment.bash"
new_env: $

Notice the new prompt. Now let's try to create a new recipe for Python 2.7 based on the source distribution. First let's see what recipes exist:

new_env: $ UpdateRecipes Python
  GetRecipe: Trying to get .../Files/Compile/Recipes/Python/3.1.2-r1
  GetRecipe: Trying to get http://gobolinux.org/recipe-store/Python--3.1.2-r1--recipe.tar.bz2
  GetRecipe: Downloading recipe from http://gobolinux.org/recipe-store/Python--3.1.2-r1--recipe.tar.bz2
  ...
  GetRecipe: Trying to get .../Files/Compile/LocalRecipes/Python/2.7
  GetRecipe: Trying to get http://gobolinux.org/recipe-store/Python--2.7-r1--recipe.tar.bz2
  GetRecipe: Downloading recipe from http://gobolinux.org/recipe-store/Python--2.7-r1--recipe.tar.bz2
  ...

As you see, there are quite a few. Now suppose that version 2.8 comes out (it won't!) and you need to update the recipie, this would be a good start. Note: we use the -s 2.7 to base the recipie on the last 2.7 recipe. (Python 3.0 is quite different and so there is a good chance that the recipe might not be a good starting point.) Use NewVersion for this:

new_env: $ NewVersion Python 2.8 -s 2.7
  NewVersion: Locating a recipe for Python...
  NewVersion: Found recipe for Python at .../Files/Compile/Recipes/Python/2.7-r1
  ...
  NewVersion: Creating recipe for Python 2.8 based on 2.7-r1.
  NewVersion: Checking URL 'http://python.org/ftp/python/2.8/Python-2.8.tar.bz2'...
  NewVersion: URL may be invalid. Please check.
  ...

Notice that the file version was intelegently changed. There is no Python 2.8 so this failed, but would work if there was a new version.

It is a good idea to base your recipes on previous ones so you can take advantage of special configure and compile flags required to get the program to work.

If you need to create a recipe for a completely new program, then use MakeRecipe and pass it the source file:

new_env: $ MakeRecipe http://www.python.org/ftp/python/2.7/Python-2.7.tar.bz2
   MakeRecipe: There is already a recipe for Python 2.7, at
   MakeRecipe: /phys/users/mforbes/hg_work/gobo/atlas/build/Files/Compile/Recipes/Python/2.7-r1
   MakeRecipe: Create another? [Y/n]
Y
   MakeRecipe: Creating recipe template for Python version 2.7.
   MakeRecipe: Press Enter to continue or Ctrl-C to cancel.

   MakeRecipe: Please fill up 'compileRecipeAuthor'...
   ...
   Connecting to www.python.org|82.94.164.162|:80... connected.
   ...
   Saving to: `Python-2.7.tar.bz2'
   ...
   MakeRecipe: Python 2.7 build system seems to be based on Autoconf.

Note that MakeRecipe found a recipe already: At this point you should stop and use NewVersion, but if it did not exist, then this would be the place to start.

If you are going to edit recipes and contribute them back to Gobo, then you should fill out the compileRecipeAuthor field as requested so you get credit. Gobo next downloads the source package (the download directory is linked to /tmp/gobo_archive/ so that you don't download multiple sources if you have several different environments. You can customize this.) It then inspects the package and informs that it has discovered a typical Autoconf build setup.

This creates a Python recipe for you:

new_env: $ ls Recipes/Python/2.7
   Resources  Recipe

Now we can compile this:

new_env: $ Compile Python 2.7
   Compile: Found recipe for Python 2.7
   GetRecipe: Trying to get .../gobo_install/Installs/new_env/build/Files/Compile/LocalRecipes/Python/2.7
   ...

This runs the standard configure; make; make install sequence plus some additional sym-linking etc. so that the programs are found properly by the Gobo environment. Now we can run the program:

new_env: $ python
   Python 2.7 (r27:82500, Jul  8 2010, 19:22:53)
   [GCC 4.1.2 20080704 (Red Hat 4.1.2-48)] on linux2
   Type "help", "copyright", "credits" or "license" for more information.
   >>>

Old README File: Some things have been changed.

Currently Mac OS X 10.5 Leopard and Ubuntu 9.10 Karmic Koala have been tested. Mac OS X requires MacPorts (http://www.macports.org/install.php) to bootstrap the installation.

The first step is to copy the configuration.sh.template file to a file named configuration.sh residing in the same folder as the goborootless.sh script. Alternatively, pass the -c <configuration file> option to specify a configuration file residing in a different location. You can customize the programs to be installed by editing the configuration file. The list of available programs is here: http://recipes.gobolinux.org/r/.

The next step is to cd to the folder where you want to install GoboRootless (the installer will create a folder called "GoboRootless" inside this folder) and execute goborootless.sh. Alternatively, pass the -i <install folder> option to install GoboRootless inside a specific folder.

You can override the default GoboLinux recipes by passing the -r <recipes folder> option. The recipes folder should contain your custom recipes.

Passing the -h option will print a brief help message containing a summary of the available options.

Please send any feedback to the GoboLinux Users mailing list: gobolinux-users@lists.gobolinux.org. Thanks.