Source

ccl-debian / README.source

Full commit
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
<!-- -*-markdown-*- -->

CCL FOR DEBIAN
================

You can convert this document to PDf form by installing pandoc and running

    markdown2pdf --toc README.source

for versions of pandoc prior to 1.9.1 or so, or

    pandoc README.source -o README.source.pdf

for more recent versions of pandoc.

The source for this package was obtained by a Subversion checkout from
`svn.clozure.com`, corresponding to the commands

    svn co svn://svn.clozure.com/openmcl/release/releasenum/source

where the release number should be substituted in place of
`releasenum`. For a script to build an `orig.tar.gz` from this source,
see the `get-orig-source` target in the `debian/rules` file.

This package uses `quilt` to manage all modifications to the upstream
source. Changes are stored in the source package as diffs in
`debian/patches` and applied during the build.

See `/usr/share/doc/quilt/README.source` for a detailed explanation.

The two ways to build this package
----------------------------------

This packaging is for Clozure Common Lisp (`CCL`). which is a Common
Lisp compiler. `CCL` requires itself to build. Since we cannot assume
it will be initially available in the Debian archives, this packaging
can be configured to build in two different configurations.

1.  Use upstream binaries in the `ccl-bootstrap` package. This is
convered in detail in the section "Building `CCL` using ccl-bootstrap: a
walkthrough". This results in the `ccl` binary package.

2.  Use the `ccl` package itself to build `ccl`. This may involve
using config 1 first if the `ccl` binary package is not
available. This is the same as the steps in "Building `CCL` using
ccl-bootstrap: a walkthrough", except steps 6 through 9 inclusive can
be skipped.

The detailed discussion in "Adapting the `CCL` build process for Debian"
applies to building using `ccl-bootstrap` (config 1). This also
applies to building using `ccl` (config 2), except that the installed
`ccl` is used.

These two versions of the packaging differ in three files, `control`,
`rules`, and `build_interface_databases`. To generate the version
for use with `ccl-bootstrap` (config 1), run

    BOOTSTRAP=true ./genscripts

inside the Debian directory. For the version for use with `ccl`
(config 2), use

    BOOTSTRAP=false ./genscripts

Building `CCL` using `ccl-bootstrap`: a walkthrough
-----------------------------------------------

The following is a step by step guide of how to create `orig.tar.gz`
tarballs for `ccl-bootstrap`, `ccl-ffigen`, and `ccl`, and create
binary debs from them using the Debian packaging. The `ccl-ffigen`
sources hava a patch applied against them. The `ccl` sources do not.

1.  Let's assume we are going to be building in `/usr/local/src/ccl`.

        cd /usr/local/src/ccl

2.  First, you need the `ccl-ffigen` package to be installed. Note
    that this package is very stable, and will probably not change
    often, so if a binary package is available, you can install that
    rather than rebuilding it from sources. If you wish to rebuild
    from source, the sources can be obtained

    (a) by doing

        apt-get source ccl-ffigen

    for the latest version of `ccl-ffigen`. This command will unpack
	the sources into a directory. You will need the debian
	subdirectory from this directory. Alternatively,

    (b) You can also obtain the Debian packaging directly from
    <https://bitbucket.org/faheem/ccl-ffigen-debian>.

3.  If you did (a), I will assume the `debian/` directory from the
    `ccl-ffigen` sources is copied to `/usr/local/src/ccl` as
    `ccl-ffigen-debian`. If (b), just clone the repository into
    `/usr/local/src/ccl` with

	    hg clone https://bitbucket.org/faheem/ccl-ffigen-debian.

	The next task is to create an orig tarball. Run

        ./ccl-ffigen-debian/rules get-orig-source

    This should create a `ccl-ffigen_1.2.orig.tar.gz`.

4.  Now, unpack this tarball, and copy the `ccl-ffigen-debian`
    directory into it as `debian`.

        tar zxvf ccl-ffigen_1.2.orig.tar.gz
        cd ccl-ffigen-1.2/
        cp -ar ../ccl-ffigen-debian/ debian

5.  Now build to obtain `*.dsc`, `*.debian.tar.gz`, and
    `ccl-ffigen_1.2-1_ARCH.deb` files with

        debuild -uc -us
        cd ..
        sudo dpkg -i ccl-ffigen_1.2-1_ARCH.deb

6.  Next you need the `ccl-bootstrap` to be installed. This is a
    build dependency. Using a version of `ccl-bootstrap` which is
    older than the source you are trying to compile may or may not
    work. According to upstream: "The only thing you can rely on is
    that a particular version of ccl can build itself."

    Alternatively, you can build `ccl-bootstrap` itself from source. You
    will need the Debian packaging for `ccl-bootstrap`. This can be
    obtained

    (a) by doing

        apt-get source ccl-bootstrap

    for the latest version of `ccl-bootstrap`. This command will
    unpack the sources into a directory. You will need the debian
    subdirectory from this directory. Alternatively,

    (b) You can also obtain the Debian packaging directly from
    <https://bitbucket.org/faheem/ccl-bootstrap-debian>

7.  If you did (a), I will assume the `debian/` directory from the
    `ccl-bootstrap` sources is copied to `/usr/local/src/ccl` as
    `ccl-bootstrap-debian`. If (b), just clone the repository into
    `/usr/local/src/ccl` with

	    hg clone https://bitbucket.org/faheem/ccl-bootstrap-debian.

	The next task is to create an orig tarball. First adjust
    `CCL_VERSION` or `CCL_WEB_PATH` in `debian/rules` as
    necessary. Then run

        ./ccl-bootstrap-debian/rules get-orig-source

    This should create a `ccl-bootstrap_1.8+svnxxxxx.orig.tar.gz`.

8.  Now, unpack this tarball, and copy the `ccl-bootstrap-debian`
    directory into it as `debian`.

        tar zxvf ccl-bootstrap_1.8+svnxxxxx.orig.tar.gz
        cd ccl-bootstrap-1.8+svnxxxxx/
        cp -ar ../ccl-bootstrap-debian/ debian

9.  Now build to obtain `*.dsc`, `*.debian.tar.gz`, and
    `ccl-bootstrap_1.8+svnxxxxx-x_ARCH.deb` files with

        debuild -uc -us
        cd ..
        sudo dpkg -i ccl-bootstrap_1.8+svnxxxxx-x_ARCH.deb

    Note that if the upstream revision of `ccl-bootstrap` is not the
    same as the version in the changelog, you will have to update the
    changelog with for example

	    dch -i

	and add an entry, e.g.

        ccl-bootstrap (1.8+svn15596-1) unstable; urgency=low

          * New upstream snapshot.

         -- Faheem Mitha <faheem@faheem.info>  Fri, 18 Jan 2013 03:19:46 -0500

10. Now we need to build `CCL` itself. You will need the Debian
    packaging for `CCL`. This can be obtained

	a) by doing

        apt-get source ccl

    for the latest version of `ccl`. This command will unpack the
    sources into a directory. You will need the debian subdirectory
    from this directory.

    b) You can also obtain the Debian packaging directly from
    <https://bitbucket.org/faheem/ccl-debian>.

11. If you did (a), I will assume the `debian/` directory from the
    `ccl` sources is copied to `/usr/local/src/ccl` as
    `ccl-debian`. If (b), just clone the repository into
    `/usr/local/src/ccl` with

	    hg clone https://bitbucket.org/faheem/ccl-debian.

    The next task is to create an orig tarball. First, make sure to
    run

        BOOTSTRAP=true ./genscripts

    in the `ccl-debian` directory, to generate versions of `control`
	and `rules` for building with the `ccl-bootstrap` package. Then
	adjust `CCL_VERSION` or `CCL_WEB_PATH` in `debian/rules` as
	necessary. This must match the version in `ccl-bootstrap`. Then
	run

        ./ccl-debian/rules get-orig-source

    This should create a `ccl_1.8+svnxxxxx.orig.tar.gz`.

12.  Now, unpack this tarball, and copy the `ccl-debian` directory into
    it as `debian`.

        tar zxvf ccl_1.8+svnxxxxx.orig.tar.gz
        cd ccl-1.8+svnxxxxx/
        cp -ar ../ccl-debian/ debian

13.  Now build to obtain `*.dsc, `*.debian.tar.gz`, and
    `ccl-bootstrap_1.8+svnxxxxx-x_ARCH.deb` files with

        debuild -uc -us
        cd ..
        sudo dpkg -i ccl_1.8-1_ARCH.deb

    Note as before that if the upstream revision of `ccl` is not the
    same as the version in the changelog, you will have to update the
    changelog with `dch -i`.

14. Finally, we can build a version of `ccl` using the `ccl` package
    itself as build dependency. We first change the `control` and
    `rules` files to build-depend on `ccl`. Then we build `ccl` again.

        cd ccl-1.8+svn15596/debian/
		BOOTSTRAP=false ./genscripts
        debuild -uc -us
        cd ..
        sudo dpkg -i ccl_1.8+svnxxxxx-x_ARCH.deb

Adapting the CCL build process for Debian
-----------------------------------------

In this section, we describe details of the build process that
probably only concern someone who is looking to modify the build
process.

`CCL` requires itself to build. However, at the time of writing, `CCL`
is not in the Debian archive. Therefore, this requires some special
care, as follows.

`CCL` heap images are available from the `CCL` Subversion
repository. The package `ccl-bootstrap` packages a heap image for each
arch. The heap images are located in `/usr/lib/ccl-bootstrap` on
installation, and are used to compile the `CCL` source.

To build `CCL` we proceed as follows.

(a) Build the lisp image.

(b) Build the interface databases, using the newly built lisp image, and
the heap image from `ccl-bootstrap`. See more details about this below
in "Building the interface databases".

(c) Build `CCL` itself, using those interface databases, the newly
built lisp image, and the heap image from `ccl-bootstrap`. See more
details about this in "Building the interface databases".

We have to build the lisp image first, because the `ccl-bootstrap`
package does not include the lisp image, but only the heap image, and
both the lisp image and the heap image are required to run the
compiler to build the interface databases.

We go through these steps in detail in the subsections below.

### Building the lisp image ###

Debian asks that packages be compiled with hardening flags; see
<http://wiki.debian.org/HardeningWalkthrough>. Therefore the lisp kernel
needs to be compiled with hardening flags, which means compiling the
lisp kernel by calling the makefile directly, as documented in
<file:///usr/share/doc/ccl/ccl-documentation.html#Building-the-kernel>.

### Building the interface databases ###

#### Overview: ####

`CCL` requires interface databases to build. See
<file:///usr/share/doc/ccl/ccl-documentation.html#The-Interface-Database>
for an overview, and
<file:///usr/share/doc/ccl/ccl-documentation.html#The-Interface-Translator>
and <http://trac.clozure.com/ccl/wiki/CustomFramework> for details on
how to rebuild.

Two things should be noted at the outset. First, the default `CCL`
upstream build as documented in
<file:///usr/share/doc/ccl/ccl-documentation.html#Building-Everything>
does not rebuild the interface databases. The interface databases are
shipped as binary files. However, Debian policy requires all binaries
to be built from source.

Second, though this not documented anywhere that I am aware of, the
interface databases do not need to be present for `CCL` to build
them. Therefore neither the `orig.tar.gz` for either `ccl` or
`ccl-bootstrap` ship the interface databases. Note, however, to build
the heap image, at least a portion of the interface database (namely,
that corresponding to the list in
`{x86-headers/x86-headers64}/libc/C/populate.sh)` need to be
present. I have not attempted to determine exactly which symbols are
needed for this build.

#### Details: ####

To build the interface databases, we first require the Foreign
Function Interface GENerator, `ffigen`. This reads header files, and
produces `ffi` files. The `ffigen` program is in the `ccl-ffigen`
package, which needs to be built first. The build process also uses
the shell script `h-to-ffi.sh`, which is included in `ccl-ffigen`.

The build of the interface databases is structured as follows. Here
`a -> b` means `a calls b`.

    debian/rules -> debian/build_interface_databases
                 -> ${CCL_INTERFACE_DB}/headername/C/populate.sh
                 ->  h-to-ffi.sh -> ffigen

Here, `build_interface_databases` is a shell script that loops over
the directories containing the `populate.sh` scripts and calls them in
turn. some of these directories may be omitted because of build
problems. If so, that is documented in `build_interface_databases`.

First, we call `build_interface_databases` as follows within
debian/rules.

	export HEADER_TMPDIR=$(CURDIR)/debian/tmp/tmp; export CCL_KERNEL=$(CCL_KERNEL);
    export CCL_INTERFACE_DB=$(CCL_INTERFACE_DB); export CURDIR=$(CURDIR);
    export CFLAGS=""; ./debian/build_interface_databases

Going through these exported variables one at a time:

`HEADER_TMPDIR` is the location for the temporary header files created
by the script `h-to-ffi.sh`. See further discussion about this below.

`CCL_KERNEL` is the name of the `CCL` Lisp kernel for a given
architecture.

`CCL_INTERFACE_DB` is the variable which is set in `rules` which
corresponds to the top level directory containing the `populate.sh`
scripts, eg. `x86-headers`.

`CURDIR` is the top level of the source directory.

`CFLAGS` is reset to an empty string because the hardening flags were
being passed to `ffigen` which didn't recognize them.  Upstream uses
`CFLAGS` to pass options to `ffigen`.

An issue is that the upstream `populate.sh` files contain header files
with hardwired absolute pathnames. These pathnames do not always match
the location of these headers on Debian. More seriously, since Debian
has recently switched to multiarch, many of these header files are
only found in paths that are architecture specific. So, they vary
depending on the architecture. Therefore, hardwiring the path names
does not work well.

First, note that the way the interface directories are built,
disregarding the details of the scripts above, is that first `ffigen`
is called, taking the absolute path to a header file as its sole
argument. It then parses this header file to a `ffi` text file, which
contains lisp expressions corresponding to the information in that ffi
file. These `ffi` files is then converted by the lisp function
`ccl::parse-standard-ffi-files` to database cdb files.

The current workaround I am using was motivated by noting that
`ffigen` (which is based on `GCC`) is able to use a search path
(presumably by using the preprocessor as `GCC` itself does) to find
the absolute path for headers that are `#include`d inside the main
header file (the one passed as an argument to `ffigen`). (Of course,
when headers are `#include`d, they specify a path relative to their
include directory.)  This search takes place among all the directory
paths that are passed via `-I` or `-isystem`. One can enable this
mechanism for the top level header file by placing it inside a
temporary header file generated for that sole purpose. For example,
given the `jni.h` header with upstream hardwired path, say
`/usr/lib/jvm/java-openjdk/include/jni,h`, which may be incorrect, and
is, on Debian, one can create a temporary header file containing only
`#include <jni.h>`. `ffigen` will then search among specified paths to
find this header. The patch
`debian/patches/ffigen4.source.h-to-ffi.sh.patch` in `ccl-ffigen`
adds logic to handle this to `h-to-ffi.sh`. The idea of adding the
logic to `h-to-ffi.sh` was suggested by Steve Langasek on #multiarch
on OFTC. I don't know why `ffigen` doesn't accept relative header
paths as arguments, and search for them in the specified directories,
as then this workaround would be unnecessary.

### Building CCL ###

Now that we have built the interface databases, we can proceed to the
final step of building the `CCL` heap image. The line

    export CCL_DEFAULT_DIRECTORY=$(CURDIR)

in `debian/rules` is important in this context, since this sets the
`CCL_DEFAULT_DIRECTORY` environmental variable to the top level of the
package sources. `CCL` uses this variable to set the value of the
`CCL` logical host to the location of these sources. This tells the
`CCL` compiler being used to look for the sources in that
location. See also Section 4.6.2
(<file:///usr/share/doc/ccl/ccl-documentation.html#Pathnames>) in the
`CCL` manual.

We build `CCL` using the methods documented in
<file:///usr/share/doc/ccl/ccl-documentation.html#Building-the-kernel>
and
<file:///usr/share/doc/ccl/ccl-documentation.html#Building-the-heap-image>
We first build the bootstrapping imaage, then the `fasl` files and
finally the heap image in separate steps.

 -- Faheem Mitha \<faheem@faheem.info\>, Tue,  29 Jan 2012