Source

ocaml-core / base / sexplib / doc / README.tex

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
\documentclass[a4paper]{article}

\usepackage{hevea}

%BEGIN LATEX
\usepackage{natbib}
%END LATEX

\newcommand{\mail}{\mailto{opensource@janestreet.com}}
\newcommand{\homeurl}{http://www.janestreet.com}
\newcommand{\athome}[2]{\ahref{\homeurl/#1}{#2}}
\newcommand{\www}{\athome{}{Markus Mottl}}

\newcommand{\ocaml}{\ahref{http://www.ocaml.org}{OCaml}}

\newcommand{\myhref}[1]{\ahref{#1}{#1}}
\newcommand{\ocsrc}[2]{\athome{ocaml/#1}{#2}}
\newcommand{\myocsrc}[1]{\athome{ocaml/#1}{#1}}

\newcommand{\janeshort}
  {\ahref{http://www.janestreet.com}{Jane Street Holding, LLC}}

\newcommand{\trow}[2]{\quad #1 \quad&\quad #2 \quad\\}
\newcommand{\trowl}[2]{\trow{#1}{#2}\hline}

%BEGIN LATEX
\newcommand{\theyear}{\number\year}
%END LATEX

\title{README: library ``Sexplib''}
\author{
  Copyright \quad (C) \quad \theyear \quad \janeshort \quad\\
  Author: Markus Mottl
}
\date{New York, 2012-04-20}

% DOCUMENT
\begin{document}
\maketitle
\section{Directory contents}
\begin{center}
\begin{tabular}{|c|c|}
\hline
\trowl{CHANGES}{History of code changes}
\trowl{COPYRIGHT}{Notes on copyright}
\trow{INSTALL}{Short notes on compiling and}
\trowl{}{installing the library}
\trowl{LICENSE}{``GNU LESSER GENERAL PUBLIC LICENSE''}
\trowl{LICENSE.Tywith}{License of Tywith, from which Sexplib is derived}
\trowl{Makefile}{Top Makefile}
\trowl{OCamlMakefile}{Generic Makefile for OCaml-projects}
\trowl{OMakefile}{Ignore this file}
\trowl{README.txt}{This file}
\trowl{lib/}{OCaml-library for S-expression conversions}
\trowl{lib\_test/}{Test applications for the Sexplib-library}
\end{tabular}
\end{center}

\section{What is ``Sexplib''?}

This library contains functionality for parsing and pretty-printing
S-expressions.  In addition to that it contains a preprocessing module for
Camlp4, which can be used to automatically generate code from type definitions
for efficiently converting OCaml-values to S-expressions and vice versa.
In combination with the parsing and pretty-printing functionality this frees
users from having to write their own I/O-routines for datastructures they
define.  Possible errors during automatic conversions from S-expressions
to OCaml-values are reported in human-readable ways with exact location
information.  Another module in this library allows you to extract and
replace sub-expressions in S-expressions.

\section{How can you use it?}

Make sure you have installed the \verb=type_conv= package on your system,
too.  It should be obtainable at the same site as \verb=sexplib=.\\
\\
The API (.mli-files) in the \verb=sexplib= library directory is
fully documented.  Module \verb=Sexp= contains all I/O-functions for
S-expressions, module \verb=Conv= helper functions for converting
OCaml-values of standard types to S-expressions.  Module \verb=Path=
supports sub-expression extraction and substitution.\\
\\
Module \verb=pa_sexp_conv.ml= contains the extensions for the
Camlp4-preprocessor.  It adds the new construct \verb=with sexp=
(and \verb=with sexp_of= and \verb=with of_sexp=, which are implied by
the first).  When using this construct right after a type definition,
function definitions will be generated automatically, which perform
S-expression conversions.\\
\\
E.g.\ given the following type definition:

\begin{verbatim}
  type t = A | B
  with sexp
\end{verbatim}

The above will generate the functions \verb=sexp_of_t= and
\verb=t_of_sexp=.  The preprocessor also supports automatic addition
of conversion functions to signatures.  Just add \verb=with sexp= to
the type in a signature, and the appropriate function signatures will
be generated.\\
\\
Converters for standard types (int, list, Hashtbl.t, etc.) become visible to
the macro-generated code by opening the standard module before their first
use in a type definition.  Users will therefore usually want to place the
following at the top of their files:

\begin{verbatim}
  open Sexplib.Std
\end{verbatim}

See the file \verb=lib_test/conv_test.ml= for example usage.  It also
demonstrates how to extract and substitute sub-expressions.

\subsection{Compiling and linking}

To compile a file you will have to add a preprocessing flag to the
compiler invocation, e.g. for foo.ml:

\begin{verbatim}
  ocamlc -pp "camlp4o -I {path to type_conv} \
    -I {path to sexplib} pa_type_conv.cmo pa_sexp_conv.cmo" \
    -I {path to sexplib} foo.ml
\end{verbatim}

If you are using OCamlMakefile, just put the following line at the
top of the file, assuming you have installed both \verb=type_conv= and
\verb=sexplib= with ocamlfind.  The comment must start at the beginning of
the line, and you must not break lines (here broken for readability only):

\begin{verbatim}
  (*pp camlp4o -I `ocamlfind query type_conv` \
    -I `ocamlfind query sexplib` \
    pa_type_conv.cmo pa_sexp_conv.cmo *)
\end{verbatim}

Don't forget to place the macro \verb=TYPE_CONV_PATH=, which takes a
string argument, at the top of the file to be preprocessed (see example
\verb=conv_test.ml=).  It is supposed to set the module path to the given
module file.  This is necessary, because modules may get packed at a later
stage, and error messages generated by Sexplib will refer to this location
to help pinpointing the associated type.\\
\\
In the linking stage you will only have to link with \verb=sexplib=.
E.g.\ when using OCamlMakefile just add it to the \verb=PACKS=-variable.

\section{Syntax Specification of S-expressions}

\subsection{Lexical conventions of S-expression}

Whitespace, which consists of space, newline, horizontal tab, and form feed,
is ignored unless within an OCaml-string, where it is treated according to
OCaml-conventions.  The left parenthesis opens a new list, the right one closes
it again.  Lists can be empty.  The double quote denotes the beginning and end
of a string following the lexical conventions of OCaml (see the OCaml-manual
for details).  All characters other than double quotes, left- and right
parentheses, whitespace, carriage return, and comment-introducing characters
or sequences (see next paragraph) are considered part of a contiguous string.\\
\\
A line comment is introduced using a semicolon, which comments out all
text up to the end of the next newline character.  The sequence \verb=%;=
introduces an S-expression comment.  This means that the next S-expression,
which must be syntactically correct and may be an atom (quoted or unquoted)
or list, following this two-character sequence will be ignored.  Whitespace
or other comments between this sequence and the subsequent S-expression are
ignored.  Block comments are opened with \verb=#|= and closed with \verb=|#=.
They can be nested and require that double-quotes within the block balance
and contain syntactically correct OCaml-strings, similar to quoted atoms.
These OCaml-strings may contain comment characters without causing parsing
problems.

\subsection{Grammar of S-expressions}

S-expressions are either strings (= atoms) or lists. The lists can
recursively contain further S-expressions or be empty, and must be
balanced, i.e.\ parentheses must match.

\subsection{Examples}

{\samepage
\begin{verbatim}
  this_is_an_atom_123'&^%!  ; this is a comment
  "another atom in an OCaml-string \"string in a string\" \123"

  ; empty list follows below
  ()

  ; a more complex example
  (
    (
      list in a list  ; comment within a list
      (list in a list in a list)
      42 is the answer to all questions
    )
  )
\end{verbatim}
}

\subsection{Conversion of basic OCaml-values}

Basic OCaml-values like the unit-value, integers (in all representations),
floats, strings, and booleans are represented in S-exp syntax in the
same way as in OCaml.  Strings may also appear without quotes if this
does not clash with the lexical conventions for S-expressions.

\subsection{Conversion of OCaml-tuples}

OCaml-tuples are simple lists of values in the same order as in the tuple.
E.g.:

\begin{verbatim}
  (3.14, "foo", "bar bla", 27) <===> (3.14 foo "bar bla" 27)
\end{verbatim}

\subsection{Conversion of OCaml-records}

OCaml-records are represented as lists of pairs in S-expression syntax.
Each pair consists of the name of the record field (first element),
and its value (second element).  E.g.:

\begin{verbatim}
  {
    foo = 3;
    bar = "some string";
  }

  <===>

  (
    (foo 3)
    (bar "some string")
  )
\end{verbatim}

Type specifications of records allow the use of a special type
\verb=sexp_option= which indicates that a record field should be optional.
E.g.:

\begin{verbatim}
  type t =
    {
      x : int option;
      y : int sexp_option;
    } with sexp
\end{verbatim}

The type \verb=sexp_option= is equivalent to ordinary options, but is
treated specially by the code generator.  The above would lead to the
following equivalences of values and S-expressions:

\begin{verbatim}
  {
    x = Some 1;
    y = Some 2;
  }

  <===>

  (
    (x (some 1))
    (y 2)
  )
\end{verbatim}

\noindent And:

{\samepage
\begin{verbatim}
  {
    x = None;
    y = None;
  }

  <===>

  (
    (x none)
  )
\end{verbatim}
}

Note how \verb=sexp_option= allows you to leave away record fields
that should default to \verb=None=.  It is also unnecessary (and
actually wrong) now to write down such a value as an option, i.e.\
the \verb=some=-tag must be dropped if the field should be defined.\\
\\
The types \verb=sexp_list=, \verb=sexp_array=, and \verb=sexp_bool= can be
used in ways similar to the type \verb=sexp_option=.  They assume the empty
list, empty array, and false value respectively as default values.\\
\\
More complex default values can be specified explicitly using several
constructs, e.g.:

\begin{verbatim}
  let z_test v = v > 42

  type t =
    {
      x : int with default(42);
      y : int with default(3), sexp_drop_default;
      z : int with default(3), sexp_drop_if(z_test);
    } with sexp
\end{verbatim}

\noindent The \verb=default= record field extension above is supported by the
underlying preprocessor library \verb=type_conv= and specifies the intended
default value for a record field in its argument.  Sexplib will use this
information to generate code which will set this record field to the default
value if an S-expression omits this field.  If a record is converted to an
S-expression, record fields with default values will be emitted as usual.
Specifying \verb=sexp_drop_default= will add a test for polymorphic equality
to the generated code such that a record field containing its default value
will be suppressed in the resulting S-expression.  This option requires the
presence of a default value.  \verb=sexp_drop_if= on the other hand does
not require a default.  Its argument must be a function, which will receive
the current record value.  If the result of this function is \verb=true=,
then the record field will be suppressed in the resulting S-expression.\\
\\
The above extensions can be quite creatively used together with manifest
types, functors, and first-class modules to make the emission of record
fields or the definition of their default values configurable at runtime.

\subsection{Conversion of sum types}

Constant constructors in sum types are represented as strings.
Constructors with arguments are represented as lists, the first element
being the constructor name, the rest being its arguments.  Constructors
may also be started in lowercase in S-expressions, but will always be
converted to uppercase when converting from OCaml-values.\\
\\
{\samepage
\noindent For example:

\begin{verbatim}
  type t = A | B of int * float * t with sexp

  B (42, 3.14, B (-1, 2.72, A)) <===> (B 42 3.14 (B -1 2.72 A))
\end{verbatim}

\noindent The above example also demonstrates recursion in datastructures.
}

\subsection{Conversion of variant types}

The conversion of polymorphic variants is almost the same as with
sum types.  The notable difference is that variant constructors must
always start with an either lower- or uppercase character, matching
the way it was specified in the type definition.  This is because OCaml
also distinguishes between upper- and lowercase variant constructors.
Note that type specifications containing unions of variant types are
also supported by the S-expression converter.

\subsection{Conversion of OCaml-lists and arrays}

OCaml-lists and arrays are straightforwardly represented as S-expression
lists.

\subsection{Conversion of option types}

The option type is converted like ordinary polymorphic sum types, i.e.:

\begin{verbatim}
  None        <===>  none
  Some value  <===>  (some value)
\end{verbatim}

There is a deprecated version of the syntax in which values of option
type are represented as lists in S-expressions:

\begin{verbatim}
  None        <===>  ()
  Some value  <===>  (value)
\end{verbatim}

Reading of the old-style S-expression syntax for option values is only
supported if the reference \verb=Conv.read_old_option_format= is set to
\verb=true= (currently the default, which may change soon).  A conversion
exception is raised otherwise.  The old format will be written only if
\verb=Conv.write_old_option_format= is true (also currently the default).
Reading of the new format is always supported.

\subsection{Conversion of polymorphic values}

There is nothing special about polymorphic values as long as there are
conversion functions for the type parameters.  E.g.:

\begin{verbatim}
  type 'a t = A | B of 'a with sexp
  type foo = int t with sexp
\end{verbatim}

In the above case the conversion functions will behave as if \verb=foo=
had been defined as a monomorphic version of \verb=t= with \verb='a=
replaced by \verb=int= on the right hand side.\\
\\
If a datastructure is indeed polymorphic, and you want to convert it,
you will have to supply the conversion functions for the type parameters
at runtime.  E.g.\ in the above example, if you wanted to convert a value
of type \verb='a t=, you would have to write something like this:

\begin{verbatim}
  sexp_of_t sexp_of_a v
\end{verbatim}

\noindent where \verb=sexp_of_a=, which may also be named differently in
this particular case, is a function that converts values of type \verb='a=
to an S-expression.  Types with more than one parameter require passing
conversion functions for those parameters in the order of their appearance
on the left hand side of the type definition.

\subsection{Conversion of abstract datatypes}

Of course, if you want to convert an abstract datatype to an S-expression,
you will have to roll your own conversion function, which should produce
values of type \verb=Sexp.t= directly.  If, however, you want to make
use of your abstract type within definitions of other types, make sure
that you call your conversion function appropriately: it should be in the
same scope as the typename, and must be named \verb=sexp_of_{typename}=.

\subsection{Conversion of hashtables}

Hashtables, which are abstract values in OCaml, are represented as
association lists, i.e.\ lists of key-value pairs, e.g.:

\begin{verbatim}
  ((foo 42) (bar 3))
\end{verbatim}

Reading in the above S-expression as hashtable mapping strings to
integers (\verb=(string, int) Hashtbl.t=) will map \verb="foo"= to $42$
and \verb="bar"= to $3$.\\
\\
Note that the order of elements in the list may matter, because
duplicates are kept: bindings will be inserted into the hashtable in
order of appearence.  Therefore, the last binding of a key will be the
``visible'' one, the others are ``hidden''.  See the OCaml-documentation
on hashtables for details.\\
\\
Note, too, that polymorphic equality may not hold between conversions.
You will have to use a function implementing logical equality for that
purpose.

\subsection{Conversion of opaque values}

Opaque values are ones for which we do not want to perform conversions.
This may be, because we do not have S-expression converters for them,
or because we do not want to apply them in a particular type context,
e.g.\ if the resulting S-expression should be printed out but without
superfluous information.  To prevent the preprocessor from generating
calls to converters, simply apply the qualifier \verb=sexp_opaque=
as if it were a type constructor, e.g.:

\begin{verbatim}
  type foo = int * stuff sexp_opaque with sexp
\end{verbatim}

Thus, there is no need to specify converters for type \verb=stuff=,
and if there are any, they will not be used in this particular context.
Needless to say, it is not possible to convert such an S-expression back
to the original value.  Here is an example conversion:

\begin{verbatim}
  (42, some_stuff) ===>  (42, <opaque>)
\end{verbatim}

\subsection{Conversion of exceptions}

S-expression converters for exceptions can be automatically registered
using the \verb=with sexp= macro, e.g.:

\begin{verbatim}
  module M = struct
    exception Foo of int with sexp
  end
\end{verbatim}

Such exceptions will be translated in a similar way as sum types, but
their constructor will be prefixed with the fully qualified module path
(here: \verb=M.Foo=) so as to be able to discriminate between them
without problems.\\

The user can then easily convert an exception matching the above
one to an S-expression using \verb=Sexplib.Conv.sexp_of_exn=.
User-defined conversion functions can be registered, too, by calling
\verb=Sexplib.Conv.add_exn_converter=.  This should make it very
convenient for users to catch arbitrary exceptions escaping their program
and pretty-printing them, including all arguments, as S-expressions.
The library already contains mappings for all known exceptions that can
escape functions in the OCaml standard library.

\section{I/O and type conversions}

There are multiple ways of performing I/O with S-expressions.  If exact
error locations are required when type conversions fail, S-expressions need
to be parsed with location annotations.  In most cases users may want to use
functions like e.g.\ \verb=load_sexp_conv= or \verb=load_sexp_conv_exn=, which
load S-expressions from files and convert them.  Only when conversions fail,
the file will be reparsed with annotations, which is slower, and type errors
will be reported accurately with file, line number, column, and file position.

\section{Contact information}

\noindent In the case of bugs, feature requests and similar, you can
contact us here:\\

\hspace{2ex}\mail\\

\noindent Up-to-date information concerning this library should be
available here:\\

\hspace{2ex}\homeurl/ocaml\\

Enjoy!!\\

\end{document}
Tip: Filter by directory path e.g. /media app.js to search for public/media/app.js.
Tip: Use camelCasing e.g. ProjME to search for ProjectModifiedEvent.java.
Tip: Filter by extension type e.g. /repo .js to search for all .js files in the /repo directory.
Tip: Separate your search with spaces e.g. /ssh pom.xml to search for src/ssh/pom.xml.
Tip: Use ↑ and ↓ arrow keys to navigate and return to view the file.
Tip: You can also navigate files with Ctrl+j (next) and Ctrl+k (previous) and view the file with Ctrl+o.
Tip: You can also navigate files with Alt+j (next) and Alt+k (previous) and view the file with Alt+o.