Source

semantic / semantic.texi

   1
   2
   3
   4
   5
   6
   7
   8
   9
  10
  11
  12
  13
  14
  15
  16
  17
  18
  19
  20
  21
  22
  23
  24
  25
  26
  27
  28
  29
  30
  31
  32
  33
  34
  35
  36
  37
  38
  39
  40
  41
  42
  43
  44
  45
  46
  47
  48
  49
  50
  51
  52
  53
  54
  55
  56
  57
  58
  59
  60
  61
  62
  63
  64
  65
  66
  67
  68
  69
  70
  71
  72
  73
  74
  75
  76
  77
  78
  79
  80
  81
  82
  83
  84
  85
  86
  87
  88
  89
  90
  91
  92
  93
  94
  95
  96
  97
  98
  99
 100
 101
 102
 103
 104
 105
 106
 107
 108
 109
 110
 111
 112
 113
 114
 115
 116
 117
 118
 119
 120
 121
 122
 123
 124
 125
 126
 127
 128
 129
 130
 131
 132
 133
 134
 135
 136
 137
 138
 139
 140
 141
 142
 143
 144
 145
 146
 147
 148
 149
 150
 151
 152
 153
 154
 155
 156
 157
 158
 159
 160
 161
 162
 163
 164
 165
 166
 167
 168
 169
 170
 171
 172
 173
 174
 175
 176
 177
 178
 179
 180
 181
 182
 183
 184
 185
 186
 187
 188
 189
 190
 191
 192
 193
 194
 195
 196
 197
 198
 199
 200
 201
 202
 203
 204
 205
 206
 207
 208
 209
 210
 211
 212
 213
 214
 215
 216
 217
 218
 219
 220
 221
 222
 223
 224
 225
 226
 227
 228
 229
 230
 231
 232
 233
 234
 235
 236
 237
 238
 239
 240
 241
 242
 243
 244
 245
 246
 247
 248
 249
 250
 251
 252
 253
 254
 255
 256
 257
 258
 259
 260
 261
 262
 263
 264
 265
 266
 267
 268
 269
 270
 271
 272
 273
 274
 275
 276
 277
 278
 279
 280
 281
 282
 283
 284
 285
 286
 287
 288
 289
 290
 291
 292
 293
 294
 295
 296
 297
 298
 299
 300
 301
 302
 303
 304
 305
 306
 307
 308
 309
 310
 311
 312
 313
 314
 315
 316
 317
 318
 319
 320
 321
 322
 323
 324
 325
 326
 327
 328
 329
 330
 331
 332
 333
 334
 335
 336
 337
 338
 339
 340
 341
 342
 343
 344
 345
 346
 347
 348
 349
 350
 351
 352
 353
 354
 355
 356
 357
 358
 359
 360
 361
 362
 363
 364
 365
 366
 367
 368
 369
 370
 371
 372
 373
 374
 375
 376
 377
 378
 379
 380
 381
 382
 383
 384
 385
 386
 387
 388
 389
 390
 391
 392
 393
 394
 395
 396
 397
 398
 399
 400
 401
 402
 403
 404
 405
 406
 407
 408
 409
 410
 411
 412
 413
 414
 415
 416
 417
 418
 419
 420
 421
 422
 423
 424
 425
 426
 427
 428
 429
 430
 431
 432
 433
 434
 435
 436
 437
 438
 439
 440
 441
 442
 443
 444
 445
 446
 447
 448
 449
 450
 451
 452
 453
 454
 455
 456
 457
 458
 459
 460
 461
 462
 463
 464
 465
 466
 467
 468
 469
 470
 471
 472
 473
 474
 475
 476
 477
 478
 479
 480
 481
 482
 483
 484
 485
 486
 487
 488
 489
 490
 491
 492
 493
 494
 495
 496
 497
 498
 499
 500
 501
 502
 503
 504
 505
 506
 507
 508
 509
 510
 511
 512
 513
 514
 515
 516
 517
 518
 519
 520
 521
 522
 523
 524
 525
 526
 527
 528
 529
 530
 531
 532
 533
 534
 535
 536
 537
 538
 539
 540
 541
 542
 543
 544
 545
 546
 547
 548
 549
 550
 551
 552
 553
 554
 555
 556
 557
 558
 559
 560
 561
 562
 563
 564
 565
 566
 567
 568
 569
 570
 571
 572
 573
 574
 575
 576
 577
 578
 579
 580
 581
 582
 583
 584
 585
 586
 587
 588
 589
 590
 591
 592
 593
 594
 595
 596
 597
 598
 599
 600
 601
 602
 603
 604
 605
 606
 607
 608
 609
 610
 611
 612
 613
 614
 615
 616
 617
 618
 619
 620
 621
 622
 623
 624
 625
 626
 627
 628
 629
 630
 631
 632
 633
 634
 635
 636
 637
 638
 639
 640
 641
 642
 643
 644
 645
 646
 647
 648
 649
 650
 651
 652
 653
 654
 655
 656
 657
 658
 659
 660
 661
 662
 663
 664
 665
 666
 667
 668
 669
 670
 671
 672
 673
 674
 675
 676
 677
 678
 679
 680
 681
 682
 683
 684
 685
 686
 687
 688
 689
 690
 691
 692
 693
 694
 695
 696
 697
 698
 699
 700
 701
 702
 703
 704
 705
 706
 707
 708
 709
 710
 711
 712
 713
 714
 715
 716
 717
 718
 719
 720
 721
 722
 723
 724
 725
 726
 727
 728
 729
 730
 731
 732
 733
 734
 735
 736
 737
 738
 739
 740
 741
 742
 743
 744
 745
 746
 747
 748
 749
 750
 751
 752
 753
 754
 755
 756
 757
 758
 759
 760
 761
 762
 763
 764
 765
 766
 767
 768
 769
 770
 771
 772
 773
 774
 775
 776
 777
 778
 779
 780
 781
 782
 783
 784
 785
 786
 787
 788
 789
 790
 791
 792
 793
 794
 795
 796
 797
 798
 799
 800
 801
 802
 803
 804
 805
 806
 807
 808
 809
 810
 811
 812
 813
 814
 815
 816
 817
 818
 819
 820
 821
 822
 823
 824
 825
 826
 827
 828
 829
 830
 831
 832
 833
 834
 835
 836
 837
 838
 839
 840
 841
 842
 843
 844
 845
 846
 847
 848
 849
 850
 851
 852
 853
 854
 855
 856
 857
 858
 859
 860
 861
 862
 863
 864
 865
 866
 867
 868
 869
 870
 871
 872
 873
 874
 875
 876
 877
 878
 879
 880
 881
 882
 883
 884
 885
 886
 887
 888
 889
 890
 891
 892
 893
 894
 895
 896
 897
 898
 899
 900
 901
 902
 903
 904
 905
 906
 907
 908
 909
 910
 911
 912
 913
 914
 915
 916
 917
 918
 919
 920
 921
 922
 923
 924
 925
 926
 927
 928
 929
 930
 931
 932
 933
 934
 935
 936
 937
 938
 939
 940
 941
 942
 943
 944
 945
 946
 947
 948
 949
 950
 951
 952
 953
 954
 955
 956
 957
 958
 959
 960
 961
 962
 963
 964
 965
 966
 967
 968
 969
 970
 971
 972
 973
 974
 975
 976
 977
 978
 979
 980
 981
 982
 983
 984
 985
 986
 987
 988
 989
 990
 991
 992
 993
 994
 995
 996
 997
 998
 999
1000
1001
1002
1003
1004
1005
1006
1007
1008
1009
1010
1011
1012
1013
1014
1015
1016
1017
1018
1019
1020
1021
1022
\input texinfo   @c -*-texinfo-*-
@c
@c $Id$
@c
@setfilename semantic.info
@settitle Semantic parsing for Emacs

@ifinfo
@format
START-INFO-DIR-ENTRY
* semantic: (semantic).       Semantic Parsing for Emacs
END-INFO-DIR-ENTRY
@end format
@end ifinfo

@titlepage
@sp 10
@center @titlefont{Semantic}
@vskip 0pt plus 1 fill
Copyright @copyright{} 1999, 2000 Eric M. Ludlam
@end titlepage

@node Top, Install, (dir), (dir)
@comment  node-name,  next,  previous,  up

Semantic is a program for Emacs which includes, at its core, a lexer,
and a compiler compiler (bovinator).  Additional tools include a
bnf->semantic table converter, example tables, and a speedbar tool.

The core utility is the ``semantic bovinator'' which has similar
behaviors as yacc or bison.  Since it is not designed to be as feature
rich as these tools, it uses the term ``bovine'' for cow, a lesser
cousin of the yak and bison.

In it's current state, the bovinator is both fast (because it uses emacs
tricks) and slow (it is recursive, and not table (array) based).  If you
have experience with compiler compilers (bison, yacc, etc) your help to
make these updates would be greatly appreciated.

Because the bovinator is recursive, you may need to add the following to
your .emacs file:

@code{(setq max-specpdl-size 1000)}

@menu
* Install::			Installing semantic.
* Lexing::                      Setting up the lexer for your language.
* Bovinating::                  Setting up the parser for your language.
* BNF conversion::              Using the BNF converter to make tables.
* Compiling::                   Running the bovinator on a source file.
* Debugging::                   Using the bovine table debugger.
* Non-Terminals::               How to use the nonterminal stream.
* Utilities::                   Utilities for querying the nonterminal stream.
@end menu

@node Install, Lexing, Top, Top
@chapter Installation

To install semantic, untar the distribution into a subdirectory, such as
@file{/usr/share/emacs/site-lisp/semantic-#.#}.  Next, add the following
lines into your individual @file{.emacs} file, or into
@file{site-lisp/site-start.el}.

@example
(add-to-list 'load-path "/path/to/semantic")

(require 'semantic-c)
(require 'semantic-el)
(require 'semantic-make)
(add-hook 'speedbar-load-hook (lambda () (require 'semantic-sb)))
(autoload 'semantic-bnf-mode "semantic-bnf" "Mode for Bovine Normal Form." t)
(add-to-list 'auto-mode-alist '("\\.bnf$" . semantic-bnf-mode))
@end example

@node Lexing, Bovinating, Install, Top
@chapter Preparing your language for Lexing

In order to reduce a source file into a token list, it must first be
converted into a token stream.  Tokens are syntactic elements such as
whitespace, symbols, strings, lists, and punctuation.

The lexer uses the major-mode's syntax table for conversion.  As long as
that is set up correctly (along with the important
@code{comment-start-skip} variable) the lexer should already work for
your language.

There are three ways to extend the flexer.

@defvar semantic-flex-extensions
For a given language buffer, this is an associatoin list.  The key is a
regular expression used to match text in the buffer.  The data element
is a function which will be called when that text is encountered.

The function should return a list of the form:
@example
(TYPE START . END)
@end example
nil is also valid in case the text does not satisfy some other criterion.

TYPE can be any type of symbol, as long as it doesn't occur as a
nonterminal in the language definition.
@end defvar

@defvar semantic-flex-syntax-modifications
Updates to the syntax table for a given buffer.
These changes are active only while this file is being flexed.
This is a list where each element is of the form:
@example
  (CHAR CLASS) 
@end example

Where CHAR is the char passed to `modify-syntax-entry',
and CLASS is the string also passed to `modify-syntax-entry' to define
what class of syntax CHAR is.

@example
  (setq semantic-flex-syntax-modifications '((?. "_"))
@end example

Will convert the period @asis{.} to be a symbol constituant.  (ie, if
filenames are prevalent as in Makefiles.)
@end defvar

@defvar semantic-flex-enable-newlines
When flexing, report 'newlines as syntactic elements.
Useful for languages where the newline is a special case terminator.
@end defvar

@node Bovinating, BNF conversion, Lexing, Top
@chapter Preparing a bovine table for your language

When converting a source file into a nonterminal token stream
(parse-tree) it is important to specify rules to accomplish this.  The
rules are stored in the buffer local variable
@code{semantic-toplevel-bovine-table}.

While it is certainly possible to write this table yourself, it is most
likely you will want to use the BNF converter.  @xref{BNF conversion}.
This is an easier method for specifying your rules.  You will still need
to specify a variable in your language for the table, however.  A good
rule of thumb is to call it @code{langauge-toplevel-bovine-table} if it
part of the language, or @code{semantic-toplevel-language-bovine-table}
if you donate it to the semantic package.

When initializing a major-mode for your language, you will set the
variable @code{semantic-toplevel-bovine-table} to the contents of your
language table.  @code{semantic-toplevel-bovine-table} is always buffer
local.

Since it is important to know the format of the table when debugging
@xref{Debugging}, you should still attempt to understand the basics of
the table.

Please see the documentation for the variable
@code{semantic-toplevel-bovine-table} for details on it's format.

* fix this *

@node BNF conversion, Compiling, Bovinating, Top
@chapter Using the BNF converter to make bovine tables

The BNF converter takes a file in "Bovine Normal Form" which is similar
to "Backus-Naur Form".  If you have ever used yacc or bison, you will
find it similar.  The BNF form used by semantic, however, does not
include token precedence rules.

It is important to have an Emacs Lisp file with a variable ready to take
the output of your table @xref{Bovinating}.  Also, make sure that the
file @file{semantic-bnf} is loaded.  Give your language file the
extension @file{.bnf} and you are ready.

The comment character is @asis{#}.

When you want to test your file, use the keyboard shortcut @kbd{C-c C-c}
to parse the file, generate the variable, and load the new definition
in.  It will then use the settings specified above to determine what to
do.

Make sure that you create the variable specified in the
@code{%parsetable} token before trying to convert the bnf file.  A
simple definition like this should be sufficient.

@example
(defvar semantic-toplevel-lang-bovine-table
   nil
   "Table for use with semantic for parsing LANG.")
@end example

The BNF file has two sections.  The first is the settings section, and
the second is the language definition, or list of semantic rules.

@menu
* Settings::
* Rules::
* Optional Lambda Expression::
* Examples::
@end menu

@node Settings, Rules, BNF conversion, BNF conversion
@section Settings

A setting is a keyword starting with a @asis{%}.  (This syntax is taken
from yacc and bison. @xref{(bison)}.)

There are several settings that can be made in the settings section.
They are:

@table @code
@item %start <nonterminal>
Will let you specify an alternative to @code{bovine-toplevel}.  (See
below)
@item %outputfile <filename>
Required.  Specifies the file into which this files output is stored.
@item %parsetable <lisp-variable-name>
Required.  Specifies a lisp variable into which the output is stored.
@item %keywordtable <lisp-variable-name>
Required if there are @code{%token} keywords.
Specifies a lisp variable into which the output of a keyword table is
stored.  This obarray is used to turn symbols into keywords when applicable.
@item %token <name> "<text>"
Optional.  Specify a new token @var{NAME}.  This is added to a lexical
keyword list using @var{TEXT}.  The symbol is then converted into a new
lexical terminal.  This requires that the @code{%keywordtable} specified
variable is available in the file specified by @code{%outputfile}.
@item %token <name> type "<text>"
Optional.  Specify a new token @var{NAME}.  It is made from an existing
lexical token of type @var{TYPE}.  @var{TEXT} is a string which will be
matched explicitly.  @var{NAME} can be used in match rules as though they were
flex tokens, but are converted back to @var{TYPE} "text" internally.
@item %languagemode <lisp-function-name>
Optional.  Specifies the Emacs major mode associated with the language
being specified.  When the language is converted, all buffers of this
mode will get the new table installed.
@item %quotemode backquote
Optional.  Specifies how symbol quoting is handled in the Optional
Lambda Expressions. (See below)
@item %setupfunction <lisp-function-name>
Name of a function into which setup code is to be inserted.
@item %( <lisp-expression> )%
Specify setup code to be inserted into the @code{%setupfunction}.
It will be inserted between two specifier strings, or added to
the end of the function.
@end table

When working inside @code{%( ... )%} tokens, any lisp expression can be
entered which will be placed inside the setup function.  In general, you
probably want to set variables that tell Semantic and related tools how
the language works.

Here are some variable that control how different programs will work
with your language.

@defvar semantic-flex-depth
Default flexing depth.
This specifies how many lists to create tokens in.
@end defvar

@defvar semantic-flex-extensions
Buffer local extensions to the lexical analyzer.
This should contain an alist with a key of a regex and a data element of
a function.  The function should both move point, and return a lexical
token of the form ( TYPE START .  END).  nil is also a valid return.
@end defvar

@defvar semantic-flex-syntax-modifications
Updates to the syntax table for this buffer.
These changes are active only while this file is being flexed.
This is a list where each element is of the form:
  (CHAR CLASS) 
Where CHAR is the char passed to `modify-syntax-entry',
and CLASS is the string also passed to `modify-syntax-entry' to define
what class of syntax CHAR is.
@end defvar

@defvar semantic-flex-enable-newlines
When flexing, report 'newlines as syntactic elements.
Useful for languages where the newline is a special case terminator.
Only set this on a per mode basis, not globally.
@end defvar

@defvar semantic-ignore-comments
Default comment handling.
t means to strip comments when flexing.  Nil means to keep comments
as part of the token stream.
@end defvar

@defvar semantic-symbol->name-assoc-list
Association between symbols returned, and a string.
The string is used to represent a group of objects of the given type.
It is sometimes useful for a language to use a different string
in place of the default, even though that language will still
return a symbol.  For example, Java return's includes, but the
string can be replaced with `Imports'.
@end defvar

@defvar semantic-case-fold
Value for `case-fold-search' when parsing.
@end defvar

@defvar semantic-expand-nonterminal
Function to call for each returned Non-terminal.
Return a list of non-terminals derived from the first argument, or nil
if it does not need to be expanded.
@end defvar

@defvar semantic-override-table
Buffer local semantic function overrides alist.
These overrides provide a hook for a `major-mode' to override specific
behaviors with respect to generated semantic toplevel nonterminals and
things that these non-terminals are useful for.
Each element must be of the form: (SYM . FUN)
where SYM is the symbol to override, and FUN is the function to
override it with.

Available override symbols:

@multitable {@code{abbreviate-nonterminal}} {(token & nosnarf)} {Find token in buffer}
@item SYBMOL                 @tab PARAMETERS        @tab DESCRIPTION
@item find-dependency        @tab (token)           @tab Find the dependency file
@item find-nonterminal       @tab (token & parent)  @tab Find token in buffer.
@item find-documentation     @tab (token & nosnarf) @tab Find doc comments.
@item abbreviate-nonterminal @tab (token & parent)  @tab Return summery string.
@item summerize-nonterminal  @tab (token & parent)  @tab Return summery string.
@item prototype-nonterminal  @tab (token)           @tab Return a prototype string.
@item prototype-file         @tab (buffer)          @tab Return a file in which prototypes are placed
@end multitable
 	                                           
Parameters mean:

@table @code
@item &
Following parameters are optional
@item buffer
The buffer in which a token was found.
@item token
The nonterminal token we are doing stuff with
@item parent
If a TOKEN is stripped (of positional infomration) then this will be the
parent token which should have positional information in it.
@end table

@end defvar

@defvar semantic-dependency-include-path
Defines the include path used when searching for files.
This should be a list of directories to search which is specific to
the file being included.
This variable can also be set to a single function.  If it is a
function, it will be called with one arguments, the file to find as a
string, and  it should return the full path to that file, or nil.
@end defvar

@defvar imenu-create-index-function
The function to use for creating a buffer index.

It should be a function that takes no arguments and returns an index
of the current buffer as an alist.

Simple elements in the alist look like @samp{(INDEX-NAME . INDEX-POSITION)}.
Special elements look like @samp{(INDEX-NAME INDEX-POSITION FUNCTION ARGUMENTS...)}.
A nested sub-alist element looks like (INDEX-NAME SUB-ALIST).
The function @code{imenu--subalist-p} tests an element and returns t
if it is a sub-alist.

This function is called within a @code{save-excursion}.

The variable is buffer-local.
@end defvar


These are specific to the document tool.

@table @code
@item document-comment-start
Comment start string.
@item document-comment-line-prefix
Comment prefix string.  Used at the beginning of each line.
@item document-comment-end
Comment end string.
@end table

@node Rules, Optional Lambda Expression, Settings, BNF conversion
@section Rules

Writing the rules should be very similar to bison for basic syntax.
Each rule is of the form

@example
RESULT : MATCH1 (optional-lambda-expression)
       | MATCH2 (optional-lambda-expression)
       ;
@end example

@var{RESULT} is a non-terminal, or a token synthisized in your grammar.
@var{MATCH} is a list of elements that are to be matched if @var{RESULT}
is to be made.  The optional lambda expression is a list containing
simplified rules for concocting the parse tree.

In bison, each time an element of a @var{MATCH} is found, it is
"shifted" onto the parser stack.  (The stack of matched elements.)  When
all of @var{MATCH1}'s elements have been matched, it is "reduced" to
@var{RESULT}.  @xref{(bison)Algorithm}.

The first @var{RESULT} written into your language specification should
be @code{bovine-toplevel}, or the symbol specified with @code{%start}.
When starting a parse for a file, this is the default token iterated
over.  You can use any token you want in place of @code{bovine-toplevel}
if you specify what that nonterminal will be with a @code{%start} token
in the settings section.

@var{MATCH} is made up of symbols and strings.  A symbol such as
@code{foo} means that a syntactic token of type @code{foo} must be
matched.  A string in the mix means that the previous symbol must have
the additional constraint of exactly matching it.  Thus, the
combination:

@example
  symbol "moose"
@end example

means that a symbol must first be encountered, and then it must
@code{string-match "moose"}.  Be especially careful to remember that the
string is a regular expression.  The code:

@example
  punctuation "."
@end example

will match any punctuation.

For the above example in bison, a LEX rule would be used to create a new
token @var{MOOSE}.  In this case, the @var{MOOSE} token would appear.
For the bovinator, this task was mixed into the language definition to
simplify implementation, though Bison's technique is more efficient.

To make a symbol match explicity for keywords, for example, you can use
the @code{%token} command in the settings section to create new symbols.

@example
%token MOOSE "moose"

find_a_moose: MOOSE
            ;
@end example

will match "moose" explicitly, unlike the previous example where moose
need only appear in the symbol.  This is because "moose" will be
converted to @var{MOOSE} in the lexical analysis stage.  Thus the symbol
@var{MOOSE} won't be available any other way.

If we specify our token in this way:

@example
%token MOOSE symbol "moose"

find_a_moose: MOOSE
            ;
@end example

then @code{MOOSE} will match the string "moose" explicitly, but it won't
do so at the lexical level, allowing use of the text "moose" in other
forms of regular expressions.

@node Optional Lambda Expression, Examples, Rules, BNF conversion
@section Optional Lambda Expressions

The OLE (Optional Lambda Expression) is converted into a bovine lambda
@xref{Bovinating}. This lambda has special short-cuts to simplify
reading the emacs bnf definition.  An OLE like this:

@example
 ( $1 )
@end example

results in a lambda return which consists entirely of the string
or object found by matching the first (zeroeth) element of match.
An OLE like this:

@example
( ,(foo $1) )
@end example

executes `foo' on the first argument, and then splices it's return
into the return list whereas:

@example
( (foo $1) )
@end example

executes foo, and that is placed in the return list.

Here are other things that can appear inline:
@table @code
@item $1
the first object matched.
@item ,$1
the first object spliced into the list (assuming it is a list from a
non-terminal)
@item '$1
the first object matched, placed in a list.  ie ( $1 )
@item foo
the symbol foo (exactly as displayed)
@item (foo)
a function call to foo which is stuck into the return list.
@item ,(foo)
a function call to foo which is spliced into the return list.
@item '(foo)
a function call to foo which is stuck into the return list in a list.
@item (EXPAND $1 nonterminal depth)
a list starting with EXPAND performs a recursive parse on the token
passed to it (represented by $1 above.)  The semantic list is a common
token to expand, as there are often interesting things in the list.
The @var{nonterminal} is a symbol in your table which the bovinator will
start with when parsing.  @var{nonterminal}'s definition is the same as
any other nonterminal.  @var{depth} should be at least 1 when
descending into a semantic list.
@item (EXPANDFULL $1 nonterminal depth)
is like EXPAND, except that the parser will iterate over
@var{nonterminal} until there are no more matches.  (The same way the
parser iterates over @code{bovine-toplevel}. This lets you have
much simpler rules in this specific case, and also lets you have
positional information in the returned tokens, and error skipping.
@end table

If the symbol @code{%quotemode backquote} is specified, then use
@code{,@@} to splice a list in, and @code{,} to evaluate the expression.
This lets you send @code{$1} as a symbol into a list instead of having
it expanded inline.

@node Examples, , Optional Lambda Expression, BNF conversion
@section Examples

The rule:

@example
SYMBOL : symbol
@end example

is equivalent to

@example
SYMBOL : symbol
         ( $1 )
@end example

which, if it matched the string "A", would return

@example
( "A" )
@end example

If this rule were used like this:

@example
ASSIGN: SYMBOL punctuation "=" SYMBOL
        ( $1 $# )
@end example

it would match "A=B", and return

@example
( ("A") ("B") )
@end example

The letters A and B come back in lists because SYMBOL is a nonterminal,
not an actual lexical element.

to get a better result with nonterminals, use @asis{,} to splice lists
in like this;

@example
ASSIGN: SYMBOL punctuation "=" SYMBOL
        ( ,$1 ,$3 )
@end example

which would return

@example
( "A" "B" )
@end example

@node Compiling, Debugging, BNF conversion, Top
@chapter Compiling a language file with the bovinator

From a program you can use the function @code{semantic-bovinate-toplevel}.
This function takes one optional parameter specifying if the cache
should be refreshed.  By default, the cached results of the last parse
are always used.  Specifying that the cache should be checked will cause
it to be flushed if it is out of date.

Another function you can use is @code{semantic-bovinate-nonterminal}.
This command takes a token stream returned by the function
@code{semantic-flex} followed by a DEPTH (as above).  This takes an
additional optional argument of NONTERMINAL which is the nonterminal in
your table it is to start parsing with.

@deffn Command bovinate
As a user, you can use @code{bovinate} which runs the
previously mentioned command, and displays the returned tokens in a
buffer.  This is a great tool for testing tables as you develop them.
@end deffn

@deffn Command semantic-clear-toplevel-cache
When a buffer is parsed, the results are cached in a local variable in
the parsed buffer.  This command will clear the cache so that the parser
is called a second time.
@end deffn

@node Debugging, Non-Terminals, Compiling, Top
@chapter Debugging a bovine table.

To debug a language is a two step process.  The first tells emacs where
to find the source code to the bovine table.  The second is to run the
debugger from a language file that uses the table.

@deffn Command semantic-bovinate-debug-set-table
Finds a variable definition under point.  Sets this to be the table used
when debugging a language definition.
@end deffn

@deffn Command bovinate-debug
Start parsing the current buffer in debug mode.  Uses the definition set
with @code{semantic-bovinate-debug-set-table}.  Use space to move to the
next definition.  It will hightlight the current semantic token in the
source buffer, and the current match symbol.

@key{C-g} halts the debugger.
@end deffn

@deffn Command bovinate-create-reference
Create a reference parse table.  A reference table can be used to
identify differences between parsing runs.
@end deffn

@deffn Command bovinate-reference-compare
Bovinates the current buffer with a reference list.
Throws an error for the first difference, with details as to what was
broken.
@end deffn

@node Non-Terminals, Utilities, Debugging, Top
@chapter Using the Non-Termianal Stream

The list of nontermianls returned from the parser is always language
dependent, but should follow some simple conventions.  A generic entry
in the stream should be of the form

@example
("NAME" type-symbol ["TYPE"] ... "DOCSTRING" OVERLAY)
@end example

In this case, @var{NAME} is the identifier being declared.
@var{type-symbol} is a symbol representing the type of identifier being
declared.  @var{TYPE} is a string containing the text of the type of
this identifier, or nil if there is no typing information.
@refill

Additional information follows TYPE which is specific to the type of
identifier being created.  The last two entries are @var{docstring},
some documentation associated with this class, and @var{overlay}, which
describe the bounds of this definition.  @var{overlay} is an overlay
object in Emacs, and an extent in XEmacs.
@refill

Some predefined nonterminals that can be used are:

@example
("NAME" variable "TYPE" CONST DEFAULT-VALUE MODIFIERS [OPTSUFFIX]
        "DOCSTRING" OVERLAY)
@end example

The definition of a variable, or constant.  @var{CONST} is a boolean
representing if this variable is considered a constant.
@var{DEFAULT-VALUE} can be something apropriate such a a string, or list
of parsed elements.  @var{MODIFIERS} are details about a variable that
are not covered in the @var{TYPE} field.  @var{OPTSUFFIX} is an optional
field specifying traling modifiers such as array dimentions or bit
fields. @var{DOCSTRING} is optional.
@refill

@example
("NAME" function "TYPE" ( ARG-LIST ) MODIFIERS [THROWS] 
        "DOCSTRING" OVERLAY)
@end example

A function/procedure definition.
@var{ARG-LIST} is a list of variable definitions.
@var{THROWS} is an optional argument for functions or methods in languages
that support typed signal throwing.
@var{DOCSTRING} is optional.

@example
("NAME" type "TYPE" ( PART-LIST ) ( PARENTS ) MODIFIERS "DOCSTRING" OVERLAY)
@end example

A type definition.
@var{TYPE} of a type could be anything, such as (in C) struct, union, typedef,
or class.
@var{PART-LIST} is only useful for structs that have multiple individual parts.
         (It is recommended that these be variables, functions or types).
@var{PARENTS} is strictly for classes where there is inheritance.

@example
("FILE" include SYSTEM "DOCSTRING" OVERLAY)
@end example

In C, an #include statement.  In elisp, a require statement.
Indicates additional locations of sources or definitions.
@var{SYSTEM} is true if this include is part of a set of system includes.

@example
("NAME" package DETAIL "DOCSTRING" OVERLAY)
@end example

In Emacs Lisp, a `provide' statement.  @var{DETAIL} might be an associated
file name.
@refill

@node Utilities,  , Non-Terminals, Top
@chapter Utilities

There are many utilities which can be used in programs that use the
nonterminal stream generated by the bovinator.  These utilities
generally require that the current buffer belongs to the stream being
analyzied.

Some of these utils will use EDE if it is available.  EDE is a tool that
tracks how program language files relate to eachother.

@menu
* Nonterminal queryies::        Get info about one nonterminal token.
* Nonterminal streams::         Get info from a stream of nonterminals.
* Nonterminal completion::      Read nonterminal names in the minibuffer.
* Override methods::            Methods that are more language specific.
@end menu

@node Nonterminal queryies, Nonterminal streams, Utilities, Utilities
@section Nonterminal queryies

These functions all take a single nonterminal (a list), and return some
element from that list.

@defun semantic-token-p token
Return non-@code{nil} if @var{TOKEN} is most likely a semantic token.
@end defun

@defun semantic-token-token token
Retrieve from @var{TOKEN} the token identifier.
ie, the symbol @code{'variable}, @code{'function}, @code{'type}, or other.
@end defun

@defun semantic-token-name token
Retrieve the name of @var{TOKEN}.
@end defun

@defun semantic-token-docstring token &optional buffer
Retrieve the documentation of @var{TOKEN}.
Optional argument @var{BUFFER} indicates where to get the text from.
If not provided, then only the @var{POSITION} can be provided.
@end defun

@defun semantic-token-overlay token
Retrieve the overlay associated with @var{TOKEN}.  The overlay is used
to store the extent of TOKEN.
@end defun

@defun semantic-token-extent token
Retrieve the extent (@var{START} @var{END}) of @var{TOKEN}.
@end defun

@defun semantic-token-start token
Retrieve the start location of @var{TOKEN}.
@end defun

@defun semantic-token-end token
Retrieve the end location of @var{TOKEN}.
@end defun

@defun semantic-token-type token
Retrieve the type of @var{TOKEN}.
@end defun

@defun semantic-token-type-parts token
Retrieve the parts of the type @var{TOKEN}.
@end defun

@defun semantic-token-type-parent token
Retrieve the parent of the type @var{TOKEN}.
@end defun

@defun semantic-token-type-modifiers token
Retrieve the non-type modifiers of type @var{TOKEN}.
@end defun

@defun semantic-token-function-args token
Retrieve the arguments of the function @var{TOKEN}.
@end defun

@defun semantic-token-function-modifiers token
Retrieve the non-type modifiers of the function @var{TOKEN}.
@end defun

@defun semantic-token-function-throws token
Retrieve the throws signale of the function @var{TOKEN}.
This is an optional field, and returns @code{nil} if it doesn't exist.
@end defun

@defun semantic-token-variable-const token
Retrieve the status of constantness from the variable @var{TOKEN}.
@end defun

@defun semantic-token-variable-default token
Retrieve the default value of the variable @var{TOKEN}.
@end defun

@defun semantic-token-variable-modifiers token
Retrieve extra non-type modifiers for the variable @var{TOKEN}.
@end defun

@defun semantic-token-include-system token
Retrieve the flag indicating if the include @var{TOKEN} is a sysmtem include.
@end defun


@node Nonterminal streams, Nonterminal completion, Nonterminal queryies, Utilities
@section Nonterminal streams

These functions take some key, and returns information found inside the
nonterminal stream returned by the bovinator.  Some will return one
token (the first matching item found.)  Others will return a list of all
items matching a given criterion.


@defun semantic-find-nonterminal-by-name name streamorbuffer
Find a nonterminal @var{NAME} within @var{STREAMORBUFFER}.  @var{NAME} is a string.
@end defun

@defun semantic-find-nonterminal-by-position position streamorbuffer &optional nomedian
Find a nonterminal covinging @var{POSITION} within @var{STREAMORBUFFER}.
@var{POSITION} is a number, or marker.  If @var{NOMEDIAN} is non-@code{nil}, don't do
the median calculation, and return nil.
@end defun

@defun semantic-find-innermost-nonterminal-by-position position streamorbuffer &optional nomedian
Find a list of nonterminals covering @var{POSITION} within
@var{STREAMORBUFFER}.  @var{POSITION} is a number, or marker.  If
@var{NOMEDIAN} is non-@code{nil}, don't do the median calculation, and
return nil.  This function will find the topmost item, and recurse until
no more details are available of findable.
@end defun

@defun semantic-find-nonterminal-by-overlay positionormarker buffer
Find all nonterminals covering @var{POSITIONORMARKER} by using overlays.
If @var{POSITIONORMARKER} is nil, use the current point.  Optional
@var{BUFFER} is used if @var{POSITIONORMARKER} is a number, otherwise
the current buffer is used.  This finds all tokens covering the
specified position by checking for all overlays covering the current
spot.  They are then sorted from largest to smallest via the start
location.
@end defun

@defun semantic-find-nonterminal-by-overlay-in-region start end buffer
Find all nonterminals which exist in whole or in part between
@var{START} and @var{END}.
Uses overlays to determine positin.
Optional @var{BUFFER} argument specifies the buffer to use.
@end defun

@defun semantic-current-nonterminal
Return the current nonterminal in the current buffer.
If there are more than one in the same location, return the
smallest token.
@end defun

@defun semantic-find-nonterminal-by-token token streamorbuffer
Find all nonterminals with a token @var{TOKEN} within @var{STREAMORBUFFER}.
@var{TOKEN} is a symbol.
@end defun

@defun semantic-find-nonterminal-standard streamorbuffer
Find all nonterminals in @var{STREAMORBUFFER} which define simple token types.
@end defun

@defun semantic-find-nonterminal-by-type type streamorbuffer
Find all nonterminals with type @var{TYPE} within @var{STREAMORBUFFER}.
@var{TYPE} is a string.
@end defun

@defun semantic-find-nonterminal-by-function function streamorbuffer
Find all nonterminals which @var{FUNCTION} match within @var{STREAMORBUFFER}.
@var{FUNCTION} must return non-@code{nil} if an element of @var{STREAM} will be included
in the new list.
@end defun

@defun semantic-find-nonterminal-by-function-first-match function streamorbuffer
Find the first nonterminal which @var{FUNCTION} match within @var{STREAMORBUFFER}.
@var{FUNCTION} must return non-@code{nil} if an element of @var{STREAM} will be included
in the new list.
@end defun

@defun semantic-recursive-find-nonterminal-by-name name buffer
Recursivly find the first occurance of @var{NAME}.
Start search with @var{BUFFER}.  Recurse through all dependencies till found.
The return item is of the form (@var{BUFFER} @var{TOKEN}) where @var{BUFFER} is the buffer
in which @var{TOKEN} (the token found to match @var{NAME}) was found.
@end defun


@node Nonterminal completion, Override methods, Nonterminal streams, Utilities
@section Nonterminal completion

These functions provide ways reading the names of items in a buffer with
completion.


@defun semantic-read-symbol prompt &optional default stream filter
Read a symbol name from the user for the current buffer.
@var{PROMPT} is the prompt to use.
Optional arguments:
@var{DEFAULT} is the default choice.  If no default is given, one is read
from under point.
@var{STREAM} is the list of tokens to complete from.
@var{FILTER} is provides a filter on the types of things to complete.
@var{FILTER} must be a function to call on each element.  (See
@end defun

@defun semantic-read-variable prompt &optional default stream
Read a variable name from the user for the current buffer.
@var{PROMPT} is the prompt to use.
Optional arguments:
@var{DEFAULT} is the default choice.  If no default is given, one is read
from under point.
@var{STREAM} is the list of tokens to complete from.
@end defun

@defun semantic-read-function prompt &optional default stream
Read a function name from the user for the current buffer.
@var{PROMPT} is the prompt to use.
Optional arguments:
@var{DEFAULT} is the default choice.  If no default is given, one is read
from under point.
@var{STREAM} is the list of tokens to complete from.
@end defun

@defun semantic-read-type prompt &optional default stream
Read a type name from the user for the current buffer.
@var{PROMPT} is the prompt to use.
Optional arguments:
@var{DEFAULT} is the default choice.  If no default is given, one is read
from under point.
@var{STREAM} is the list of tokens to complete from.
@end defun


@node Override methods,  , Nonterminal completion, Utilities
@section Override methods

These functions are called `override methods' because they provide
generic behaviors, which a given language can override.  For example,
finding a dependency file in Emacs lisp can be done with the
`locate-library' command (which overrides the default behavior.)  In C,
a dependency can be found by searching a generic search path which can
be passed in via a variable.


@defun semantic-find-dependency token
Find the filename represented from @var{TOKEN}.
@var{TOKEN} may be a stripped element, in which case @var{PARENT} specifies a
parent token that has positinal information.
Depends on @code{semantic-dependency-include-path} for searching.  Always searches
`.' first, then searches additional paths.


@defvar semantic-dependency-include-path
Defines the include path used when searching for files.
This should be a list of directories to search which is specific to
the file being included.
This variable can also be set to a single function.  If it is a
function, it will be called with one arguments, the file to find as a
string, and  it should return the full path to that file, or nil.
@end defvar


@end defun


@defun semantic-find-nonterminal token &optional parent
Find the location of @var{TOKEN}.
@var{TOKEN} may be a stripped element, in which case @var{PARENT} specifies a
parent token that has position information.
Different behaviors are provided depending on the type of token.
For example, dependencies (includes) will seek out the file that is
depended on, and functions will move to the specified definition.
@end defun

@defun semantic-find-documentation token
Find documentation from @var{TOKEN} and return it as a clean string.
@var{TOKEN} might have @var{DOCUMENTATION} set in it already.  If not, there may be
some documentation in a comment preceeding TOKEN's definition which we
cal look for.  When appropriate, this can be overridden by a language specific
enhancement.
@end defun

@defun semantic-summerize-nonterminal token &optional parent
Summerize @var{TOKEN} in a reasonable way.
Optional argument @var{PARENT} is the parent type if @var{TOKEN} is a detail.
@end defun

@defun semantic-prototype-nonterminal token
Return a prototype for @var{TOKEN}.
This functin must be overloaded, though it need not be used.
@end defun

@defun semantic-prototype-file buffer
Return a file in which prototypes belonging to @var{BUFFER} should be placed.
Default behavior (if not overriden) looks for a token specifying the
prototype file, or the existence of an @var{EDE} variable indicating which
file prototypes belong in.
@end defun


@bye
Tip: Filter by directory path e.g. /media app.js to search for public/media/app.js.
Tip: Use camelCasing e.g. ProjME to search for ProjectModifiedEvent.java.
Tip: Filter by extension type e.g. /repo .js to search for all .js files in the /repo directory.
Tip: Separate your search with spaces e.g. /ssh pom.xml to search for src/ssh/pom.xml.
Tip: Use ↑ and ↓ arrow keys to navigate and return to view the file.
Tip: You can also navigate files with Ctrl+j (next) and Ctrl+k (previous) and view the file with Ctrl+o.
Tip: You can also navigate files with Alt+j (next) and Alt+k (previous) and view the file with Alt+o.