Source

ccl-debian / README.source

# -*- markdown -*-

You can convert this document to PDf form by installing pandoc and running

    markdown2pdf README.source

CCL FOR DEBIAN
================

The source for this package was obtained by a Subversion checkout from
svn.clozure.com, corresponding to the commands

    svn co svn://svn.clozure.com/openmcl/release/releasenum/source

where the release number should be substituted in place of
'releasenum'. For a script to build an orig.tar.gz from this source,
see the `get-orig-source` target in the `debian/rules` file.

This package uses quilt to manage all modifications to the upstream
source. Changes are stored in the source package as diffs in
debian/patches and applied during the build.

See /usr/share/doc/quilt/README.source for a detailed explanation.

A build process walkthrough
---------------------------

The following is a step by step guide of how to create orig.tar.gz
tarballs for ccl and ccl-bootstrap, and create binary debs from them
using the Debian packaging. Since the CCL sources only have trivial
sources applied against them, this should be a straightforward matter,

1.  Let's assume we are going to be building in `/usr/local/src/ccl`.

        cd /usr/local/src/ccl

2.  First you need ccl-bootstrap, which is a build dependency. Using a
    version of ccl-bootstrap which is older than the source you are
    trying to compile may or may not work. According to upstream: "The
    only thing you can rely on is that a particular version of ccl can
    build itself."

    Alternatively, you can build ccl-bootstrap itself from source. You
    will need the Debian packaging for ccl-bootstrap. This can be
    obtained, for example, by doing

        apt-get source ccl-bootstrap

    for the latest version packaged for CCL. This command will unpack
    the sources into a directory. You will need the debian
    subdirectory from this directory.

    You may also be able to obtain the Debian packaging directly from
    a repository.

3.  I'll assume the debian/ directory for ccl-bootstrap is copied to
    /usr/local/src/ccl as ccl-bootstrap-debian. The next task is to
    create an orig tarball. First adjust CCL_VERSION or CCL_WEB_PATH
    in debian/rules as necessary. Then run

        ./ccl-debian/rules get-orig-source

    This should create a ccl-bootstrap_1.8.orig.tar.gz.

4.  Now, unpack this tarball, and copy the ccl-debian directory into
    is as debian.

        tar zxvf ccl-bootstrap_1.8.orig.tar.gz
        cd ccl-bootstrap-1.8/
        cp -r ../ccl-bootstrap-debian/ debian

5.  Now you can either rebuild to obtain new *.dsc and *.debian.tar.gz
    files with

        debuild -uc -us
        cd ..
        sudo dpkg -i ccl-bootstrap_1.8-1_ARCH.deb

    or just build the binary files.

        debuild binary
        cd ..
        sudo dpkg -i ccl-bootstrap_1.8-1_ARCH.deb

6.  Now we need to build CCL itself. You will need the Debian
    packaging for CCL. This can be obtained, for example, by doing

        apt-get source ccl

    for the latest version packaged for CCL. This command will unpack
    the sources into a directory. You will need the debian
    subdirectory from this directory.

    You may also be able to obtain the Debian packaging directly from
    a repository.

7.  I'll assume the debian/ directory for ccl is copied to
    /usr/local/src/ccl as ccl-debian. The next task is to create an
    orig tarball. First adjust CCL_VERSION or CCL_WEB_PATH in
    debian/rules as necessary. This must match the version in
    ccl-bootstrap. Then run

        ./ccl-debian/rules get-orig-source

    This should create a ccl_1.8.orig.tar.gz.

8)  Now, unpack this tarball, and copy the ccl-debian directory into is
    as debian.

        tar zxvf ccl_1.8.orig.tar.gz
        cd ccl-1.8/
        cp -r ../ccl-debian/ debian

9)  Now you can either rebuild to obtain new *.dsc and *.debian.tar.gz
    files with

        debuild -uc -us
        cd ..
        sudo dpkg -i ccl_1.8-1_ARCH.deb

    or just build the binary files.

        debuild binary
        cd ..
        sudo dpkg -i ccl_1.8-1_ARCH.deb

Adapting the CCL build process for Debian
-----------------------------------------

In this section, we describe details that probably only concern
someone who is looking to modify the build process.

When building CCL as described in step (9) of "Updating the CCL
package for new releases", we can proceed as follows.

(a) Build the interface headers. See more details about this below in
"Building the interface headers".

(b) Build CCL itself, using those interface headers and bootstrap
binaries. See more details about this in "Building CCL using the
interface headers and bootstrap binaries".

### Building the interface headers ###

#### Overview: ####

CCL requires interface headers to build. See
file:///usr/share/doc/ccl/ccl-documentation.html#The-Interface-Database
for an overview, and
file:///usr/share/doc/ccl/ccl-documentation.html#The-Interface-Translator
and http://trac.clozure.com/ccl/wiki/CustomFramework for details on
how to rebuild.

Two things should be noted at the outset. First, the default CCL
upstream build as documented in
file:///usr/share/doc/ccl/ccl-documentation.html#Building-Everything
does not rebuild the interface databases. The interface databases are
shipped as binary files. However, Debian policy requires all binaries
to be built from source.

Second, though this not documented anywhere that I am aware of, the
interface databases do not need to be present for CCL to build
them. Therefore neither the orig.tar.gz for either ccl or
ccl-bootstrap ship the interface databases. Note, however, to build
the heap image, at least a portion of the interface database (namely,
that corresponding to the list in
`{x86-headers/x86-headers64}/libc/C/populate.sh)` need to be
present. I have not attempted to determine exactly which symbols are
needed for this build.

#### Details: ####

To build the interface databases, we first require the Foreign
Function Interface GENerator, ffigen. This reads header files, and
produces ffi files.

The sources for building the ffigen are included in the orig.tar.gz
tarball (see item (7) in "Updating the CCL package for new releases").
When the tarball is unpacked, these sources are contained in the
'ffigen4' directory.

We first build these sources to produce the `ffigen` binary. The build
process also uses the shell script `h-to-ffi.sh`. Since these files are
not installed anywhere in the system path, we have to do some tweaking
so that our build system can find them.

The build of the interface databases is structured as follows. Here
`a -> b` means `a calls b`.

    debian/rules -> debian/build_interface_headers.sh
                 -> ${CCL_INTERFACE_DB}/headername/C/populate.sh
                 ->  h-to-ffi.sh -> ffigen

Here, build_interface_headers.sh is a shell script that loops over the
directories containing the populate.sh scripts and calls them in turn.

First, we call build_interface_headers.sh as follows within
debian/rules.

    export CCL_INTERFACE_DB=$(CCL_INTERFACE_DB); export CURDIR=$(CURDIR);
    export CFLAGS=""; ./debian/build_interface_headers.sh

Going through these exported variables one at a time:

CCL_INTERFACE_DB is the variable which is set in rules which
corresponds to the top level directory containing the populate.sh
scripts, eg. x86-headers.

CURDIR is the top level of the source directory.

CFLAGS is reset to an empty string because the hardening flags were
being passed to ffigen which didn't recognize them.  Upstream uses
CFLAGS to pass options to ffigen.

The first issue is that populate.sh needs to call `h-to-ffi.sh`, but
`h-to-ffi.sh` is not in PATH. We could patch the populate.sh, but a
less invasive approach is to add `$CURDIR/ffigen4/bin` (the location
of the ffigen binary) to PATH in `build_interface_headers.sh`.

A similar issue was that h-to-ffi.sh needs to calls ffigen, but again
ffigen is not in PATH. So we patch `h-to-ffi.sh` to set FFIGEN_DIR to
$CURDIR/ffigen4/ffigen. FFIGEN_DIR is the directory that contains the
ffigen binary as well as some header files that ffigen uses.

The next issue is that the upstream `populate.sh` files contain header
files with hardwired absolute pathnames. These pathnames do not always
match the location of these headers on Debian. More seriously, since
Debian has recently switched to multiarch, many of these header files
are only found in paths that are architecture specific. So, they vary
depending on the architecture. Therefore, hardwiring the path names
does not work well.

First, note that the way the interface directories are built,
disregarding the details of the scripts above, is that first ffigen is
called, taking the absolute path to a header file as its sole
argument. It then parses this header file to a ffi text file, which
contains lisp expressions corresponding to the information in that ffi
file. These ffi files is then converted by the lisp function
ccl::parse-standard-ffi-files to database cdb files.

The current workaround I am using was motivated by noting that ffigen
(which is based on GCC) is able to use a search path (presumably by
using the preprocessor as GCC itself does) to find the absolute path
for headers that are #included inside the main header file (the one
passed as an argument to ffigen). (Of course, when headers are
#included, they specify a path relative to their include directory.)
This search takes place among all the directory paths that are passed
via -I or -isystem. One can enable this mechanism for the top level
header file by placing it inside a temporary header file generated for
that sole purpose. For example, given the `jni.h` header with upstream
hardwired path, say `/usr/lib/jvm/java-openjdk/include/jni,h`, which may
be incorrect, and is, on Debian, one can create a temporary header
file containing only `#include <jni.h>`. ffigen will then search among
specified paths to find this header. The patch
`debian/patches/ffigen4.source.h-to-ffi.sh.patch` adds logic to handle
this to `h-to-ffi.sh`.  The idea of doing this was suggested by Steve
Langasek on #multiarch on OFTC. I don't know why ffigen doesn't accept
relative header paths as arguments, and search for them in the
specified directories, as then this workaround would be unnecessary.

### Building CCL using the interface headers and bootstrap binaries ###

CCL requires itself to build. However, at the time of writing, CCL is
not in the Debian archive. Therefore, this requires some special care,
as follows.

CCL binaries are available from the CCL Subversion repository. The
package ccl-bootstrap is packaged from these binaries. These binaries
are located in `/usr/lib/ccl-bootstrap` on installation, and are used
to compile the CCL source.

NB: The line

    export CCL_DEFAULT_DIRECTORY=$(CURDIR)

in debian/rules is important in this context, since this sets the
CCL_DEFAULT_DIRECTORY environmental variable to the location of this
package's sources, namely .../.../ccl-versionnumber. CCL uses this
variable to set the value of the CCL logical host to the location of
these sources. This tells the CCL binaries in ccl-bootstrap to compile
those sources. See also Section 4.6.2
(file:///usr/share/doc/ccl/ccl-documentation.html#Pathnames) in the
CCL manual.

We build CCL using the methods documented in
file:///usr/share/doc/ccl/ccl-documentation.html#Building-the-kernel
and
file:///usr/share/doc/ccl/ccl-documentation.html#Building-the-heap-image

Using the shortcut documented in
file:///usr/share/doc/ccl/ccl-documentation.html#Building-Everything
is not an option, because Debian asks that packages be compiled with
hardening flags; see
http://wiki.debian.org/HardeningWalkthrough. Therefore the lisp kernel
needs to be compiled with hardening flags, which means compiling the
lisp kernel by calling the makefile directly, as documented in
file:///usr/share/doc/ccl/ccl-documentation.html#Building-the-kernel.
This implies that one then needs to build the bootstrapping imaage,
fasl files and finally the heap image in separate steps.

 -- Faheem Mitha <faheem@faheem.info>, Thu,  9 Aug 2012 02:23:14 +0530