Source

semantic / semantic.texi

   1
   2
   3
   4
   5
   6
   7
   8
   9
  10
  11
  12
  13
  14
  15
  16
  17
  18
  19
  20
  21
  22
  23
  24
  25
  26
  27
  28
  29
  30
  31
  32
  33
  34
  35
  36
  37
  38
  39
  40
  41
  42
  43
  44
  45
  46
  47
  48
  49
  50
  51
  52
  53
  54
  55
  56
  57
  58
  59
  60
  61
  62
  63
  64
  65
  66
  67
  68
  69
  70
  71
  72
  73
  74
  75
  76
  77
  78
  79
  80
  81
  82
  83
  84
  85
  86
  87
  88
  89
  90
  91
  92
  93
  94
  95
  96
  97
  98
  99
 100
 101
 102
 103
 104
 105
 106
 107
 108
 109
 110
 111
 112
 113
 114
 115
 116
 117
 118
 119
 120
 121
 122
 123
 124
 125
 126
 127
 128
 129
 130
 131
 132
 133
 134
 135
 136
 137
 138
 139
 140
 141
 142
 143
 144
 145
 146
 147
 148
 149
 150
 151
 152
 153
 154
 155
 156
 157
 158
 159
 160
 161
 162
 163
 164
 165
 166
 167
 168
 169
 170
 171
 172
 173
 174
 175
 176
 177
 178
 179
 180
 181
 182
 183
 184
 185
 186
 187
 188
 189
 190
 191
 192
 193
 194
 195
 196
 197
 198
 199
 200
 201
 202
 203
 204
 205
 206
 207
 208
 209
 210
 211
 212
 213
 214
 215
 216
 217
 218
 219
 220
 221
 222
 223
 224
 225
 226
 227
 228
 229
 230
 231
 232
 233
 234
 235
 236
 237
 238
 239
 240
 241
 242
 243
 244
 245
 246
 247
 248
 249
 250
 251
 252
 253
 254
 255
 256
 257
 258
 259
 260
 261
 262
 263
 264
 265
 266
 267
 268
 269
 270
 271
 272
 273
 274
 275
 276
 277
 278
 279
 280
 281
 282
 283
 284
 285
 286
 287
 288
 289
 290
 291
 292
 293
 294
 295
 296
 297
 298
 299
 300
 301
 302
 303
 304
 305
 306
 307
 308
 309
 310
 311
 312
 313
 314
 315
 316
 317
 318
 319
 320
 321
 322
 323
 324
 325
 326
 327
 328
 329
 330
 331
 332
 333
 334
 335
 336
 337
 338
 339
 340
 341
 342
 343
 344
 345
 346
 347
 348
 349
 350
 351
 352
 353
 354
 355
 356
 357
 358
 359
 360
 361
 362
 363
 364
 365
 366
 367
 368
 369
 370
 371
 372
 373
 374
 375
 376
 377
 378
 379
 380
 381
 382
 383
 384
 385
 386
 387
 388
 389
 390
 391
 392
 393
 394
 395
 396
 397
 398
 399
 400
 401
 402
 403
 404
 405
 406
 407
 408
 409
 410
 411
 412
 413
 414
 415
 416
 417
 418
 419
 420
 421
 422
 423
 424
 425
 426
 427
 428
 429
 430
 431
 432
 433
 434
 435
 436
 437
 438
 439
 440
 441
 442
 443
 444
 445
 446
 447
 448
 449
 450
 451
 452
 453
 454
 455
 456
 457
 458
 459
 460
 461
 462
 463
 464
 465
 466
 467
 468
 469
 470
 471
 472
 473
 474
 475
 476
 477
 478
 479
 480
 481
 482
 483
 484
 485
 486
 487
 488
 489
 490
 491
 492
 493
 494
 495
 496
 497
 498
 499
 500
 501
 502
 503
 504
 505
 506
 507
 508
 509
 510
 511
 512
 513
 514
 515
 516
 517
 518
 519
 520
 521
 522
 523
 524
 525
 526
 527
 528
 529
 530
 531
 532
 533
 534
 535
 536
 537
 538
 539
 540
 541
 542
 543
 544
 545
 546
 547
 548
 549
 550
 551
 552
 553
 554
 555
 556
 557
 558
 559
 560
 561
 562
 563
 564
 565
 566
 567
 568
 569
 570
 571
 572
 573
 574
 575
 576
 577
 578
 579
 580
 581
 582
 583
 584
 585
 586
 587
 588
 589
 590
 591
 592
 593
 594
 595
 596
 597
 598
 599
 600
 601
 602
 603
 604
 605
 606
 607
 608
 609
 610
 611
 612
 613
 614
 615
 616
 617
 618
 619
 620
 621
 622
 623
 624
 625
 626
 627
 628
 629
 630
 631
 632
 633
 634
 635
 636
 637
 638
 639
 640
 641
 642
 643
 644
 645
 646
 647
 648
 649
 650
 651
 652
 653
 654
 655
 656
 657
 658
 659
 660
 661
 662
 663
 664
 665
 666
 667
 668
 669
 670
 671
 672
 673
 674
 675
 676
 677
 678
 679
 680
 681
 682
 683
 684
 685
 686
 687
 688
 689
 690
 691
 692
 693
 694
 695
 696
 697
 698
 699
 700
 701
 702
 703
 704
 705
 706
 707
 708
 709
 710
 711
 712
 713
 714
 715
 716
 717
 718
 719
 720
 721
 722
 723
 724
 725
 726
 727
 728
 729
 730
 731
 732
 733
 734
 735
 736
 737
 738
 739
 740
 741
 742
 743
 744
 745
 746
 747
 748
 749
 750
 751
 752
 753
 754
 755
 756
 757
 758
 759
 760
 761
 762
 763
 764
 765
 766
 767
 768
 769
 770
 771
 772
 773
 774
 775
 776
 777
 778
 779
 780
 781
 782
 783
 784
 785
 786
 787
 788
 789
 790
 791
 792
 793
 794
 795
 796
 797
 798
 799
 800
 801
 802
 803
 804
 805
 806
 807
 808
 809
 810
 811
 812
 813
 814
 815
 816
 817
 818
 819
 820
 821
 822
 823
 824
 825
 826
 827
 828
 829
 830
 831
 832
 833
 834
 835
 836
 837
 838
 839
 840
 841
 842
 843
 844
 845
 846
 847
 848
 849
 850
 851
 852
 853
 854
 855
 856
 857
 858
 859
 860
 861
 862
 863
 864
 865
 866
 867
 868
 869
 870
 871
 872
 873
 874
 875
 876
 877
 878
 879
 880
 881
 882
 883
 884
 885
 886
 887
 888
 889
 890
 891
 892
 893
 894
 895
 896
 897
 898
 899
 900
 901
 902
 903
 904
 905
 906
 907
 908
 909
 910
 911
 912
 913
 914
 915
 916
 917
 918
 919
 920
 921
 922
 923
 924
 925
 926
 927
 928
 929
 930
 931
 932
 933
 934
 935
 936
 937
 938
 939
 940
 941
 942
 943
 944
 945
 946
 947
 948
 949
 950
 951
 952
 953
 954
 955
 956
 957
 958
 959
 960
 961
 962
 963
 964
 965
 966
 967
 968
 969
 970
 971
 972
 973
 974
 975
 976
 977
 978
 979
 980
 981
 982
 983
 984
 985
 986
 987
 988
 989
 990
 991
 992
 993
 994
 995
 996
 997
 998
 999
1000
1001
1002
1003
1004
1005
1006
1007
1008
1009
1010
1011
1012
1013
1014
1015
1016
1017
1018
1019
1020
1021
1022
1023
1024
1025
1026
1027
1028
1029
1030
1031
1032
1033
1034
1035
1036
1037
1038
1039
1040
1041
1042
1043
1044
1045
1046
1047
1048
1049
1050
1051
1052
1053
1054
1055
1056
1057
1058
1059
1060
1061
1062
1063
1064
1065
1066
1067
1068
1069
1070
1071
1072
1073
1074
1075
1076
1077
1078
1079
1080
1081
1082
1083
1084
1085
1086
1087
1088
1089
1090
1091
1092
1093
1094
1095
1096
1097
1098
1099
1100
1101
1102
1103
1104
1105
1106
1107
1108
1109
1110
1111
1112
1113
1114
1115
1116
1117
1118
1119
1120
1121
1122
1123
1124
1125
1126
1127
1128
1129
1130
1131
1132
1133
1134
1135
1136
1137
1138
1139
1140
1141
1142
1143
1144
1145
1146
1147
1148
1149
1150
1151
1152
1153
1154
1155
1156
1157
1158
1159
1160
1161
1162
1163
1164
1165
1166
1167
1168
1169
1170
1171
1172
1173
1174
1175
1176
1177
1178
1179
1180
1181
1182
1183
1184
1185
1186
1187
1188
1189
1190
1191
1192
1193
1194
1195
1196
1197
1198
1199
1200
1201
1202
1203
1204
1205
1206
1207
1208
1209
1210
1211
1212
1213
1214
1215
1216
1217
1218
1219
1220
1221
1222
1223
1224
1225
1226
1227
1228
1229
1230
1231
1232
1233
1234
1235
1236
1237
1238
1239
1240
1241
1242
1243
1244
1245
1246
1247
1248
1249
1250
1251
1252
1253
1254
1255
1256
1257
1258
1259
1260
1261
1262
1263
1264
1265
1266
1267
1268
1269
1270
1271
1272
1273
1274
1275
1276
1277
1278
1279
1280
1281
1282
1283
1284
1285
1286
1287
1288
1289
1290
1291
1292
1293
1294
1295
1296
1297
1298
1299
1300
1301
1302
1303
1304
1305
1306
1307
1308
1309
1310
1311
1312
1313
1314
1315
1316
1317
1318
1319
1320
1321
1322
1323
1324
1325
1326
1327
1328
1329
1330
1331
1332
1333
1334
1335
1336
1337
1338
1339
1340
1341
1342
1343
1344
1345
1346
1347
1348
1349
1350
1351
1352
1353
1354
1355
1356
1357
1358
1359
1360
1361
1362
1363
1364
1365
1366
1367
1368
1369
1370
1371
1372
1373
1374
1375
1376
1377
1378
1379
1380
1381
1382
1383
1384
1385
1386
1387
1388
1389
1390
1391
1392
1393
1394
1395
1396
1397
1398
1399
1400
1401
1402
1403
1404
1405
1406
1407
1408
1409
1410
1411
1412
1413
1414
1415
1416
1417
1418
1419
1420
1421
1422
1423
1424
1425
1426
1427
1428
1429
1430
1431
1432
1433
1434
1435
1436
1437
1438
1439
1440
1441
1442
1443
1444
1445
1446
1447
1448
1449
1450
1451
1452
1453
1454
1455
1456
1457
1458
1459
1460
1461
1462
1463
1464
1465
1466
1467
1468
1469
1470
1471
1472
1473
1474
1475
1476
1477
1478
1479
1480
1481
1482
1483
1484
1485
1486
1487
1488
1489
1490
1491
1492
1493
1494
1495
1496
1497
1498
1499
1500
1501
1502
1503
1504
1505
1506
1507
1508
1509
1510
1511
1512
1513
1514
1515
1516
1517
1518
1519
1520
1521
1522
1523
1524
1525
1526
1527
1528
1529
1530
1531
1532
1533
1534
1535
1536
1537
1538
1539
1540
1541
1542
1543
1544
1545
1546
1547
1548
1549
1550
1551
1552
1553
1554
1555
1556
1557
1558
1559
1560
1561
1562
1563
1564
1565
1566
1567
1568
1569
1570
1571
1572
1573
1574
1575
1576
1577
1578
1579
1580
1581
1582
1583
1584
1585
1586
1587
1588
1589
1590
1591
1592
1593
1594
1595
1596
1597
1598
1599
1600
1601
1602
1603
1604
1605
1606
1607
1608
1609
1610
1611
1612
1613
1614
1615
1616
1617
1618
1619
1620
1621
1622
1623
1624
1625
1626
1627
1628
1629
1630
1631
1632
1633
1634
1635
1636
1637
1638
1639
1640
1641
1642
1643
1644
1645
1646
1647
1648
1649
1650
1651
1652
1653
1654
1655
1656
1657
1658
1659
1660
1661
1662
1663
1664
1665
1666
1667
1668
1669
1670
1671
1672
1673
1674
1675
1676
1677
1678
1679
1680
1681
1682
1683
1684
1685
1686
1687
1688
1689
1690
1691
1692
1693
1694
1695
1696
1697
1698
1699
1700
1701
1702
1703
1704
1705
1706
1707
1708
1709
1710
1711
1712
1713
1714
1715
1716
1717
1718
1719
1720
1721
1722
1723
1724
1725
1726
1727
1728
1729
1730
1731
1732
1733
1734
1735
1736
1737
1738
1739
1740
1741
1742
1743
1744
1745
1746
1747
1748
1749
1750
1751
1752
1753
1754
1755
1756
1757
1758
1759
1760
1761
1762
1763
1764
1765
1766
1767
1768
1769
1770
1771
1772
1773
1774
1775
1776
1777
1778
1779
1780
1781
1782
1783
1784
1785
1786
1787
1788
1789
1790
1791
1792
1793
1794
1795
1796
1797
1798
1799
1800
1801
1802
1803
1804
1805
1806
1807
1808
1809
1810
1811
1812
1813
1814
1815
1816
1817
1818
1819
1820
1821
1822
1823
1824
1825
1826
1827
1828
1829
1830
1831
1832
1833
1834
1835
1836
1837
1838
1839
1840
1841
1842
1843
1844
1845
1846
1847
1848
1849
1850
1851
1852
1853
1854
1855
1856
1857
1858
1859
1860
1861
1862
1863
1864
1865
1866
1867
1868
1869
1870
1871
1872
1873
1874
1875
1876
1877
1878
1879
1880
1881
1882
1883
1884
1885
1886
1887
1888
1889
1890
1891
1892
1893
1894
1895
1896
1897
1898
1899
1900
1901
1902
1903
1904
1905
1906
1907
1908
1909
1910
1911
1912
1913
1914
1915
1916
1917
1918
1919
1920
1921
1922
1923
1924
1925
1926
1927
1928
1929
1930
1931
1932
1933
1934
1935
1936
1937
1938
1939
1940
1941
1942
1943
1944
1945
1946
1947
1948
1949
1950
1951
1952
1953
1954
1955
1956
1957
1958
1959
1960
1961
1962
1963
1964
1965
1966
1967
1968
1969
1970
1971
1972
1973
1974
1975
1976
1977
1978
1979
1980
1981
1982
1983
1984
1985
1986
1987
1988
1989
1990
1991
1992
1993
1994
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
2024
2025
2026
2027
2028
2029
2030
2031
2032
2033
2034
2035
2036
2037
2038
2039
2040
2041
2042
2043
2044
2045
2046
2047
2048
2049
2050
2051
2052
2053
2054
2055
2056
2057
2058
2059
2060
2061
2062
2063
2064
2065
2066
2067
2068
2069
2070
2071
2072
2073
2074
2075
2076
2077
2078
2079
2080
2081
2082
2083
2084
2085
2086
2087
2088
2089
2090
2091
2092
2093
2094
2095
2096
2097
2098
2099
2100
2101
2102
2103
2104
2105
2106
2107
2108
2109
2110
2111
2112
2113
2114
2115
2116
2117
2118
2119
2120
2121
2122
2123
2124
2125
2126
2127
2128
2129
2130
2131
2132
2133
2134
2135
2136
2137
2138
2139
2140
2141
2142
2143
2144
2145
2146
2147
2148
2149
2150
2151
2152
2153
2154
2155
2156
2157
2158
2159
2160
2161
2162
2163
2164
2165
2166
2167
\input texinfo   @c -*-texinfo-*-
@c
@c $Id$
@c
@setfilename semantic.info
@settitle Semantic Bovinator: Parser generation for Emacs

@ifinfo
@format
START-INFO-DIR-ENTRY
* semantic: (semantic).       Semantic Parsing for Emacs
END-INFO-DIR-ENTRY
@end format
@end ifinfo

@titlepage
@sp 10
@center @titlefont{Semantic}
@vskip 0pt plus 1 fill
Copyright @copyright{} 1999, 2000, 2001 Eric M. Ludlam
@end titlepage

@node Top, Install, (dir), (dir)
@comment  node-name,  next,  previous,  up

Semantic is a program for Emacs which includes, at its core, a lexer,
and a compiler compiler (bovinator).  Additional tools include a
bnf->semantic table converter, example tables, and a speedbar tool.

The core utility is the ``semantic bovinator'' which has similar
behaviors as yacc or bison.  Since it is not designed to be as feature
rich as these tools, it uses the term ``bovine'' for cow, a lesser
cousin of the yak and bison.

In it's current state, the bovinator is both fast (because it uses Emacs
tricks) and slow (it is recursive, and not table (array) based).  If you
have experience with compiler compilers (bison, yacc, etc) your help to
make these updates would be greatly appreciated.


@menu
* Install::                     Installing semantic.
* Lexing::                      Setting up the lexer for your language.
* Bovinating::                  Setting up the parser for your language.
* BNF conversion::              Using the BNF converter to make tables.
* Compiling::                   Running the bovinator on a source file.
* Debugging::                   Debugging bovine tables
* Utilities::                   How to use the nonterminal stream.
* Current Context::             How to get the current code context.
* Tools::                       User Tools which use semantic.
* Index::                       
@end menu

@node Install, Lexing, Top, Top
@chapter Installation

To install semantic, untar the distribution into a subdirectory, such as
@file{/usr/share/emacs/site-lisp/semantic-#.#}.  Next, add the following
lines into your individual @file{.emacs} file, or into
@file{site-lisp/site-start.el}.

@example
(setq semantic-load-turn-everything-on t)
(load-file "/path/to/semantic/semantic-load.el")
@end example

If you would like to turn individual tools on or off in your init
file, skip the first line.

@node Lexing, Bovinating, Install, Top
@chapter Preparing your language for Lexing

In order to reduce a source file into a token list, it must first be
converted into a token stream.  Tokens are syntactic elements such as
whitespace, symbols, strings, lists, and punctuation.

The lexer uses the major-mode's syntax table for conversion.  As long
as that is set up correctly (along with the important
@code{comment-start} and @code{comment-start-skip} variable) the lexer
should already work for your language.

There are three ways to extend the lexer.

@defvar semantic-flex-extensions
Buffer local extensions to the lexical analyzer.
This should contain an alist with a key of a regex and a data element of
a function.  The function should both move point, and return a lexical
token of the form:

@example
( TYPE START . END)
@end example

@code{nil} is also a valid return.
@var{TYPE} can be any type of symbol, as long as it doesn't occur as a
nonterminal in the language definition.
@end defvar

@defvar semantic-flex-syntax-modifications
Updates to the syntax table for a given buffer.
These changes are active only while this file is being flexed.
This is a list where each element is of the form:
@example
(CHAR CLASS) 
@end example

Where CHAR is the char passed to `modify-syntax-entry',
and CLASS is the string also passed to `modify-syntax-entry' to define
what class of syntax CHAR is.

@example
(setq semantic-flex-syntax-modifications '((?. "_"))
@end example

Will convert the period @asis{.} to be a symbol constituant.  (ie, if
filenames are prevalent as in Makefiles.)
@end defvar

@defvar semantic-flex-enable-newlines
When flexing, report @code{'newlines} as syntactic elements.
Useful for languages where the newline is a special case terminator.
Only set this on a per mode basis, not globally.
@end defvar

@section Keywords

Another important piece of the lexer is the keyword table. @ref{Settings}.
You language will want to set up a keyword table for fast conversion of
symbol strings to language terminals.

The keywords table can also be used to store additional information
about those keywords.  The following programming functions can be useful
when examining text in a language buffer.

@defun semantic-flex-keyword-p text
Return a symbol if @var{TEXT} is a keyword in the keyword table.
@end defun

@defun semantic-flex-keyword-put text property value
For keyword @var{TEXT}, set @var{PROPERTY} to have @var{VALUE}.
@end defun

@defun semantic-flex-keyword-get text property
For the keyword found as @var{text}, get the value of @var{property}.
@end defun

@defun semantic-flex-map-keywords fun &optional property
Call function @var{FUN} on every semantic keyword.
If optional @var{PROPERTY} is non-nil call @var{FUN} only on every
keyword which have a @var{PROPERTY} value.
@var{FUN} receives a semantic keyword as argument.
@end defun

@defun semantic-flex-keywords &optional property
Return a list of semantic keywords.
If optional @var{PROPERTY} is non-nil return only keyword which have a
PROPERTY value.
@end defun

Keyword properties can be set up in a BNF file for ease of maintenance.
While examining the text in a language buffer, this can provide an easy
and quick way of storing details about text in the buffer.

@section Standard Keyword Properties

Add known properties here when they are known.

@node Bovinating, BNF conversion, Lexing, Top
@chapter Preparing a bovine table for your language

When converting a source file into a nonterminal token stream
(parse-tree) it is important to specify rules to accomplish this.  The
rules are stored in the buffer local variable
@code{semantic-toplevel-bovine-table}.

While it is certainly possible to write this table yourself, it is most
likely you will want to use the BNF converter.  @xref{BNF conversion}.
This is an easier method for specifying your rules.  You will still need
to specify a variable in your language for the table, however.  A good
rule of thumb is to call it @code{language-toplevel-bovine-table} if it
part of the language, or @code{semantic-toplevel-language-bovine-table}
if you donate it to the semantic package.

When initializing a major-mode for your language, you will set the
variable @code{semantic-toplevel-bovine-table} to the contents of your
language table.  @code{semantic-toplevel-bovine-table} is always buffer
local.

Since it is important to know the format of the table when debugging ,
you should still attempt to understand the basics of the table.

Please see the documentation for the variable
@code{semantic-toplevel-bovine-table} for details on it's format.

* add more doc here *

@node BNF conversion, Compiling, Bovinating, Top
@chapter Using the BNF converter to make bovine tables

The BNF converter takes a file in "Bovine Normal Form" which is similar
to "Backus-Naur Form".  If you have ever used yacc or bison, you will
find it similar.  The BNF form used by semantic, however, does not
include token precedence rules, and several other features needed to make
real parser generators.

It is important to have an Emacs Lisp file with a variable ready to take
the output of your table @xref{Bovinating}.  Also, make sure that the
file @file{semantic-bnf.el} is loaded.  Give your language file the
extension @file{.bnf} and you are ready.

The comment character is @asis{#}.

When you want to test your file, use the keyboard shortcut @kbd{C-c C-c}
to parse the file, generate the variable, and load the new definition
in.  It will then use the settings specified above to determine what to
do.  Use the shortcut @kbd{C-c c} to do the same thing, but spend
extra time indenting the table nicely.

Make sure that you create the variable specified in the
@code{%parsetable} token before trying to convert the bnf file.  A
simple definition like this is sufficient.

@example
(defvar semantic-toplevel-lang-bovine-table
   nil
   "Table for use with semantic for parsing LANG.")
@end example

If you use tokens (created with the @code{%token} specifier), also
make sure you have a keyword table available, like this:

@example
(defvar semantic-lang-keyword-table
   nil
   "Table for use with semantic for keywords.")
@end example

Specify the name of the keyword table with the @code{%keywordtable}
specifier.

The BNF file has two sections.  The first is the settings section, and
the second is the language definition, or list of semantic rules.

@menu
* Settings::                    
* Rules::                       
* Optional Lambda Expression::  
* Examples::                    
@end menu

@node Settings, Rules, BNF conversion, BNF conversion
@section Settings

A setting is a keyword starting with a @asis{%}.  (This syntax is taken
from yacc and bison. @xref{(bison)}.)

There are several settings that can be made in the settings section.
They are:

@table @code
@item %start <nonterminal>
Specify an alternative to @code{bovine-toplevel}.  (See below)
@item %scopestart <nonterminal>
Specify an alternitive to @code{bovine-inner-scope}.
@item %outputfile <filename>
Required.  Specifies the file into which this files output is stored.
@item %parsetable <lisp-variable-name>
Required.  Specifies a lisp variable into which the output is stored.
@item %keywordtable <lisp-variable-name>
Required if there are @code{%token} keywords.
Specifies a lisp variable into which the output of a keyword table is
stored.  This obarray is used to turn symbols into keywords when applicable.
@item %token <name> "<text>"
Optional.  Specify a new token @var{NAME}.  This is added to a lexical
keyword list using @var{TEXT}.  The symbol is then converted into a new
lexical terminal.  This requires that the @code{%keywordtable} specified
variable is available in the file specified by @code{%outputfile}.
@item %token <name> type "<text>"
Optional.  Specify a new token @var{NAME}.  It is made from an existing
lexical token of type @var{TYPE}.  @var{TEXT} is a string which will be
matched explicitly.  @var{NAME} can be used in match rules as though they were
flex tokens, but are converted back to @var{TYPE} "text" internally.
@item %put <NAME> symbol <VALUE>
@itemx %put <NAME> ( symbol1 <VALUE1> symbol2 <VALUE2> ... )
@itemx %put ( <NAME1> <NAME2>...) symbol <VALUE>
Tokens created without a type are considered keywords, and placed in a
keyword table.  Use @code{%put} to apply properties to that keyword.
@ref{Lexing}.
@item %languagemode <lisp-function-name>
@itemx %languagemode ( <lisp-function-name1> <lisp-function-name2> ... )
Optional.  Specifies the Emacs major mode associated with the language
being specified.  When the language is converted, all buffers of this
mode will get the new table installed.
@item %quotemode backquote
Optional.  Specifies how symbol quoting is handled in the Optional
Lambda Expressions. (See below)
@item %setupfunction <lisp-function-name>
Name of a function into which setup code is to be inserted.
@item %( <lisp-expression> )%
Specify setup code to be inserted into the @code{%setupfunction}.
It will be inserted between two specifier strings, or added to
the end of the function.
@end table

When working inside @code{%( ... )%} tokens, any lisp expression can be
entered which will be placed inside the setup function.  In general, you
probably want to set variables that tell Semantic and related tools how
the language works.

Here are some variables that control how different programs will work
with your language.

@defvar semantic-flex-depth
Default flexing depth.
This specifies how many lists to create tokens in.
@end defvar

@defvar semantic-flex-extensions
Buffer local extensions to the lexical analyzer.
This should contain an alist with a key of a regex and a data element of
a function.  The function should both move point, and return a lexical
token of the form:
@example
( TYPE START . END)
@end example

@code{nil} is also a valid return.
@var{TYPE} can be any type of symbol, as long as it doesn't occur as a
nonterminal in the language definition.
@end defvar

@defvar semantic-flex-syntax-modifications
Updates to the syntax table for this buffer.
These changes are active only while this file is being flexed.
This is a list where each element is of the form:
@example
(CHAR CLASS)
@end example
Where @var{CHAR} is the char passed to @dfn{modify-syntax-entry},
and @var{CLASS} is the string also passed to @dfn{modify-syntax-entry} to define
what class of syntax @var{CHAR} is.
@end defvar

@defvar semantic-flex-enable-newlines
When flexing, report @code{'newlines} as syntactic elements.
Useful for languages where the newline is a special case terminator.
Only set this on a per mode basis, not globally.
@end defvar

@defvar semantic-ignore-comments
Default comment handling.
@code{t} means to strip comments when flexing.  @code{Nil} means to keep comments
as part of the token stream.
@end defvar

@defvar semantic-symbol->name-assoc-list
Association between symbols returned, and a string.
The string is used to represent a group of objects of the given type.
It is sometimes useful for a language to use a different string
in place of the default, even though that language will still
return a symbol.  For example, Java return's includes, but the
string can be replaced with @code{Imports}.
@end defvar

@defvar semantic-case-fold
Value for @code{case-fold-search} when parsing.
@end defvar

@defvar semantic-expand-nonterminal
Function to call for each nonterminal production.
Return a list of non-terminals derived from the first argument, or @code{nil}
if it does not need to be expanded.
Languages with compound definitions should use this function to expand
from one compound symbol into several.  For example, in @var{C} the
definition
@example
int a, b;
@end example
is easily parsed into one token, but represents multiple variables.  A
functions should be written which takes this compound token and turns
it into two tokens, one for @var{A}, and the other for @var{B}.

Within the language definition (the @file{.bnf} sources), it is often
useful to set the NAME slot of a token with a list of items that
distinguish each element in the compound definition.

This list can then be detected by the function set in
@code{semantic-expand-nonterminal} to create multiple tokens.
This function has one additional duty of managing the overlays created
by semantic.  It is possible to use the single overlay in the compound
token for all your tokens, but this can pose problems identifying
all tokens covering a given definition.

Please see @file{semantic-java.el} for an example of managing overlays
when expanding a token into multiple definitions.
@end defvar

@defvar semantic-override-table
Buffer local semantic function overrides alist.
These overrides provide a hook for a `major-mode' to override specific
behaviors with respect to generated semantic toplevel nonterminals and
things that these non-terminals are useful for.
Each element must be of the form: (SYM . FUN)
where SYM is the symbol to override, and FUN is the function to
override it with.

Available override symbols:

@multitable {@code{abbreviate-nonterminal}} {(token & nosnarf)} {Find token in buffer}
@item SYBMOL                 @tab PARAMETERS        @tab DESCRIPTION
@item find-dependency        @tab (token)           @tab Find the dependency file
@item find-nonterminal       @tab (token & parent)  @tab Find token in buffer.
@item find-documentation     @tab (token & nosnarf) @tab Find doc comments.
@item abbreviate-nonterminal @tab (token & parent)  @tab Return summery string.
@item summarize-nonterminal  @tab (token & parent)  @tab Return summery string.
@item prototype-nonterminal  @tab (token)           @tab Return a prototype string.
@item prototype-file         @tab (buffer)          @tab Return a file in which prototypes are placed
@item nonterminal-children   @tab (token)           @tab Return first rate children. These are children which may contain overlays.
@item nonterminal-protection @tab (token & parent)  @tab Return protection as a symbol.
@item beginning-of-context   @tab (& point)         @tab Move to the beginning of the
					                 current context.
@item end-of-context         @tab (& point)         @tab Move to the end of the
					                 current context.
@item up-context             @tab (& point)         @tab Move up one context level.
@item get-local-variables    @tab (& point)         @tab Get local variables.
@item get-all-local-variables@tab (& point)         @tab Get all local variables.
@item get-local-arguments    @tab (& point)         @tab Get arguments to this function.

@item end-of-command         @tab                   @tab Move to the end of the current
                                                         command
@item beginning-of-command   @tab                   @tab Move to the beginning of the
                                                         current command
@item ctxt-current-symbol    @tab (& point)         @tab List of related symbols.
@item ctxt-current-assignment@tab (& point)         @tab Variable being assigned to.
@item ctxt-current-function  @tab (& point)         @tab Function being called at point.
@item ctxt-current-argument  @tab (& point)         @tab The index to the argument of
                                                         the current function the cursor
                                                         is in.
@end multitable
                                                   
Parameters mean:

@table @code
@item &
Following parameters are optional
@item buffer
The buffer in which a token was found.
@item token
The nonterminal token we are doing stuff with
@item parent
If a TOKEN is stripped (of positional infomration) then this will be the
parent token which should have positional information in it.
@end table

@end defvar

@defvar semantic-type-relation-separator-character
Character strings used to separation a parent/child relationship.
This list of strings are used for displaying or finding separators
in variable field dereferencing.  The first character will be used for
display.  In @var{C}, a type field is separated like this: ``type.field''
thus, the character is a ``.''.  In @var{C}, and additional value of ``->''
would be in the list, so that ``type->field'' could be found.
@end defvar


@defvar semantic-dependency-include-path
Defines the include path used when searching for files.
This should be a list of directories to search which is specific to
the file being included.
This variable can also be set to a single function.  If it is a
function, it will be called with one arguments, the file to find as a
string, and  it should return the full path to that file, or nil.
@end defvar

This configures Imenu to use semantic parsing.

@defvar imenu-create-index-function
The function to use for creating a buffer index.

It should be a function that takes no arguments and returns an index
of the current buffer as an alist.

Simple elements in the alist look like @samp{(INDEX-NAME . INDEX-POSITION)}.
Special elements look like @samp{(INDEX-NAME INDEX-POSITION FUNCTION ARGUMENTS...)}.
A nested sub-alist element looks like (INDEX-NAME SUB-ALIST).
The function @code{imenu--subalist-p} tests an element and returns t
if it is a sub-alist.

This function is called within a @code{save-excursion}.

The variable is buffer-local.
@end defvar


These are specific to the document tool.

@table @code
@item document-comment-start
Comment start string.
@item document-comment-line-prefix
Comment prefix string.  Used at the beginning of each line.
@item document-comment-end
Comment end string.
@end table

@node Rules, Optional Lambda Expression, Settings, BNF conversion
@section Rules

Writing the rules should be very similar to bison for basic syntax.
Each rule is of the form

@example
RESULT : MATCH1 (optional-lambda-expression)
       | MATCH2 (optional-lambda-expression)
       ;
@end example

@var{RESULT} is a non-terminal, or a token synthisized in your grammar.
@var{MATCH} is a list of elements that are to be matched if @var{RESULT}
is to be made.  The optional lambda expression is a list containing
simplified rules for concocting the parse tree.

In bison, each time an element of a @var{MATCH} is found, it is
"shifted" onto the parser stack.  (The stack of matched elements.)  When
all of @var{MATCH1}'s elements have been matched, it is "reduced" to
@var{RESULT}.  @xref{(bison)Algorithm}.

The first @var{RESULT} written into your language specification should
be @code{bovine-toplevel}, or the symbol specified with @code{%start}.
When starting a parse for a file, this is the default token iterated
over.  You can use any token you want in place of @code{bovine-toplevel}
if you specify what that nonterminal will be with a @code{%start} token
in the settings section.

@var{MATCH} is made up of symbols and strings.  A symbol such as
@code{foo} means that a syntactic token of type @code{foo} must be
matched.  A string in the mix means that the previous symbol must have
the additional constraint of exactly matching it.  Thus, the
combination:

@example
symbol "moose"
@end example

means that a symbol must first be encountered, and then it must
@code{string-match "moose"}.  Be especially careful to remember that the
string is a regular expression.  The code:

@example
punctuation "."
@end example

will match any punctuation.

For the above example in bison, a LEX rule would be used to create a new
token @var{MOOSE}.  In this case, the @var{MOOSE} token would appear.
For the bovinator, this task was mixed into the language definition to
simplify implementation, though Bison's technique is more efficient.

To make a symbol match explicity for keywords, for example, you can use
the @code{%token} command in the settings section to create new symbols.

@example
%token MOOSE "moose"

find_a_moose: MOOSE
            ;
@end example

will match ``moose'' explicitly, unlike the previous example where moose
need only appear in the symbol.  This is because ``moose'' will be
converted to @var{MOOSE} in the lexical analysis stage.  Thus the symbol
@var{MOOSE} won't be available any other way.

If we specify our token in this way:

@example
%token MOOSE symbol "moose"

find_a_moose: MOOSE
            ;
@end example

then @code{MOOSE} will match the string "moose" explicitly, but it won't
do so at the lexical level, allowing use of the text "moose" in other
forms of regular expressions.

Non symbol tokens are also alowed.  For example:

@example
%token PERIOD punctuation "."

filename : symbol PERIOD symbol
         ;
@end example

will explicitly match one period when used in the above rule.

@node Optional Lambda Expression, Examples, Rules, BNF conversion
@section Optional Lambda Expressions

The OLE (Optional Lambda Expression) is converted into a bovine lambda
@xref{Bovinating}. This lambda has special short-cuts to simplify
reading the emacs bnf definition.  An OLE like this:

@example
( $1 )
@end example

results in a lambda return which consists entirely of the string
or object found by matching the first (zeroeth) element of match.
An OLE like this:

@example
( ,(foo $1) )
@end example

executes `foo' on the first argument, and then splices it's return
into the return list whereas:

@example
( (foo $1) )
@end example

executes foo, and that is placed in the return list.

Here are other things that can appear inline:
@table @code
@item $1
the first object matched.
@item ,$1
the first object spliced into the list (assuming it is a list from a
non-terminal)
@item '$1
the first object matched, placed in a list.  ie ( $1 )
@item foo
the symbol foo (exactly as displayed)
@item (foo)
a function call to foo which is stuck into the return list.
@item ,(foo)
a function call to foo which is spliced into the return list.
@item '(foo)
a function call to foo which is stuck into the return list in a list.
@item (EXPAND $1 nonterminal depth)
a list starting with EXPAND performs a recursive parse on the token
passed to it (represented by $1 above.)  The semantic list is a common
token to expand, as there are often interesting things in the list.
The @var{nonterminal} is a symbol in your table which the bovinator will
start with when parsing.  @var{nonterminal}'s definition is the same as
any other nonterminal.  @var{depth} should be at least 1 when
descending into a semantic list.
@item (EXPANDFULL $1 nonterminal depth)
is like EXPAND, except that the parser will iterate over
@var{nonterminal} until there are no more matches.  (The same way the
parser iterates over @code{bovine-toplevel}. This lets you have
much simpler rules in this specific case, and also lets you have
positional information in the returned tokens, and error skipping.
@item (ASSOC symbol1 value1 symbol2 value2 ... )
This is used for creating an association list.  Each @var{SYMBOL} is
included in the list if the associated @var{VALUE} is non-nil.  While
the items are all listed explicitly, the created structure is an
association list of the form:
@example
( ( symbol1 . value1) (symbol2 . value2) ... )
@end example
@end table

If the symbol @code{%quotemode backquote} is specified, then use
@code{,@@} to splice a list in, and @code{,} to evaluate the expression.
This lets you send @code{$1} as a symbol into a list instead of having
it expanded inline.

@node Examples,  , Optional Lambda Expression, BNF conversion
@section Examples

The rule:

@example
SYMBOL : symbol
@end example

is equivalent to

@example
SYMBOL : symbol
         ( $1 )
@end example

which, if it matched the string "A", would return

@example
( "A" )
@end example

If this rule were used like this:

@example
ASSIGN: SYMBOL punctuation "=" SYMBOL
        ( $1 $# )
@end example

it would match "A=B", and return

@example
( ("A") ("B") )
@end example

The letters A and B come back in lists because SYMBOL is a nonterminal,
not an actual lexical element.

to get a better result with nonterminals, use @asis{,} to splice lists
in like this;

@example
ASSIGN: SYMBOL punctuation "=" SYMBOL
        ( ,$1 ,$3 )
@end example

which would return

@example
( "A" "B" )
@end example

@node Compiling, Debugging, BNF conversion, Top
@chapter Compiling a language file with the bovinator

From a program you can use the function @code{semantic-bovinate-toplevel}.
This function takes one optional parameter specifying if the cache
should be refreshed.  By default, the cached results of the last parse
are always used.  Specifying that the cache should be checked will cause
it to be flushed if it is out of date.

Another function you can use is @code{semantic-bovinate-nonterminal}.
This command takes a token stream returned by the function
@code{semantic-flex} followed by a DEPTH (as above).  This takes an
additional optional argument of NONTERMINAL which is the nonterminal in
your table it is to start parsing with.

@deffn Command bovinate &optional clear
Bovinate the current buffer.  Show output in a temp buffer.
Optional argument @var{CLEAR} will clear the cache before bovinating.
@end deffn

@deffn Command semantic-clear-toplevel-cache
Clear the toplevel bovin cache for the current buffer.
Clearing the cache will force a complete reparse next time a token
stream is requested.
@end deffn

@defun semantic-bovinate-toplevel &optional checkcache
Bovinate the entire current buffer.
If the optional argument @var{CHECKCACHE} is non-@code{nil}, then flush the cache iff
there has been a size change.
@end defun

@node Debugging, Utilities, Compiling, Top
@comment  node-name,  next,  previous,  up
@chapter Debugging

Writing language files using BNF is significantly easier than writing
then using regular expressions in a functional manner.  Debugging
them, however, can still prove challenging.

There are two ways to debug a language definition if it is not
behaving as expected.  One way is to debug against the source @file{.bnf}
file.  The second is to debug against the lisp table created from the
@file{.bnf} source, or perhaps written by hand.

If your language definition was written in BNF notation, debugging is
quite easy.  The command @code{bovinate-debug} will start you off.

@deffn Command bovinate-debug
Bovinate the current buffer and run in debug mode.
@end deffn

If you prefer debugging against the Lisp table, find the table in a
buffer, place the cursor in it, and use the command
@code{semantic-bovinate-debug-set-table} in it.

@deffn Command semantic-bovinate-debug-set-table
Set the table for the next debug to be here.
@end deffn

After the table is set, the @code{bovinate-debug} command can be run
at any time for the given language.

While debugging, two windows are visible.  One window shows the file
being parse, and the syntactic token being tested is highlighted.  The
second window shows the table being used (either in the BNF source, or
the Lisp table.) with the current rule highlighted.  The cursor will
sit on the specific match rule being tested against.

In the minibuffer, a brief summary of the current situation is
listed.  The first element is the syntactic token which is a list of
the form:

@example
(TYPE START . END)
@end example

The rest of the display is a list of all strings collected for the
currently tested rule.  Each time a new rule is entered, the list is
restarted.  Upon returning from a rule into a previous match list, the
previous match list is restored, with the production of the dependent
rule in the list.

Use @kbd{C-g} to stop debugging.  There are no commands for any
fancier types of debugging.

@node Utilities, Current Context, Debugging, Top
@comment  node-name,  next,  previous,  up
@chapter Utilities

Once a source file has been parsed, the follwing APIs can be used to
write programs that use the token stream most effectivly.

@menu
* Token Queries::               Getting info about a parsed token (nonterminal).
* Nonterminal Streams::         Working with streams of nonterminals
* Nonterminal Completion::      Completing read functions.
* Override Methods::            Language dependent functions.
* Parser Hooks::                How to know when tags change.
@end menu

@node Token Queries, Nonterminal Streams, Utilities, Utilities
@comment  node-name,  next,  previous,  up
@section Token Queries

When writing programs that use the bovinator, the following functions
are needed to find get details out of a nonterminal.

@defun semantic-token-token token
Retrieve from @var{TOKEN} the token identifier.
ie, the symbol @code{'variable}, @code{'function}, @code{'type}, or other.
@end defun

@defun semantic-token-name token
Retrieve the name of @var{TOKEN}.
@end defun

@defun semantic-token-docstring token &optional buffer
Retrieve the documentation of @var{TOKEN}.
Optional argument @var{BUFFER} indicates where to get the text from.
If not provided, then only the @var{POSITION} can be provided.
@end defun

@defun semantic-token-overlay token
Retrieve the @var{OVERLAY} part of @var{TOKEN}.
The returned item may be an overlay or an unloaded buffer representation.
@end defun

@defun semantic-token-extent token
Retrieve the extent (@var{START} @var{END}) of @var{TOKEN}.
@end defun

@defun semantic-token-start token
Retrieve the start location of @var{TOKEN}.
@end defun

@defun semantic-token-end token
Retrieve the end location of @var{TOKEN}.
@end defun

@defun semantic-token-type token
Retrieve the type of @var{TOKEN}.
@end defun

@defun semantic-token-put token property value
On @var{token}, set @var{property} to @var{value}.
@end defun

@defun semantic-token-get token property
For @var{token} get the value of @var{property}.
@end defun

@defun semantic-token-extra-spec token spec
Retrieve a specifier for the variable @var{TOKEN}.
@var{SPC} is the symbol whose modifier value to get.
This function can get specifiers from any type of @var{TOKEN}.
Do not use this function if you know what type of token you are dereferencing.
Instead, use the function specific to that token type.  It will be faster.
@end defun

@defun semantic-token-type-parts token
Retrieve the parts of the type @var{TOKEN}.
@end defun

@defun semantic-token-type-parent token
Retrieve the parent of the type @var{TOKEN}.
The return value is a list.  @var{A} value of @code{nil} means no parents.
The @dfn{car} of the list is either the parent class, or a list
of parent classes.  The @dfn{cdr} of the list is the list of
interfaces, or abstract classes which are parents of @var{TOKEN}.
@end defun

@defun semantic-token-type-parent-superclass token
Retrieve the parent superclasses of the type @var{TOKEN}.
@end defun

@defun semantic-token-type-parent-implement token
Retrieve the parent interfaces of the type @var{TOKEN}.
@end defun

@defun semantic-token-type-modifiers token
Retrieve the type modifiers for the type @var{TOKEN}.
@end defun

@defun semantic-token-type-extra-specs token
Retrieve the extra specifiers for the type @var{TOKEN}.
@end defun

@defun semantic-token-type-extra-spec token spec
Retrieve a extra specifier for the type @var{TOKEN}.
@var{SPEC} is the symbol whose modifier value to get.
@end defun

@defun semantic-token-function-args token
Retrieve the arguments of the function @var{TOKEN}.
@end defun

@defun semantic-token-function-modifiers token
Retrieve the type modifiers of the function @var{TOKEN}.
@end defun

@defun semantic-token-function-extra-specs token
Retrieve the extra specifiers of the function @var{TOKEN}.
@end defun

@defun semantic-token-function-extra-spec token spec
Retrieve a specifier for the function @var{TOKEN}.
@var{SPEC} is a symbol whose specifier value to get.
@end defun

@defun semantic-token-function-throws token
Retrieve the throws signal of the function @var{TOKEN}.
This is an optional field, and returns @code{nil} if it doesn't exist.
@end defun

@defun semantic-token-function-parent token
The parent of the function @var{TOKEN}.
A function has a parent if it is a method of a class, and if the
function does not appear in body of it's parent class.
@end defun

@defun semantic-token-variable-const token
Retrieve the status of constantness from the variable @var{TOKEN}.
@end defun

@defun semantic-token-variable-default token
Retrieve the default value of the variable @var{TOKEN}.
@end defun

@defun semantic-token-variable-modifiers token
Retrieve type modifiers for the variable @var{TOKEN}.
@end defun

@defun semantic-token-variable-extra-specs token
Retrieve extra specifiers for the variable @var{TOKEN}.
@end defun

@defun semantic-token-variable-extra-spec token spec
Retrieve a specifier value for the variable @var{TOKEN}.
@var{SPEC} is the symbol whose specifier value to get.
@end defun

@defun semantic-token-include-system token
Retrieve the flag indicating if the include @var{TOKEN} is a sysmtem include.
@end defun

For override methods that query a token, see @xref{Token Details}. 

@node Nonterminal Streams, Nonterminal Completion, Token Queries, Utilities
@section Nonterminal streams

These functions take some key, and returns information found inside the
nonterminal stream returned by the bovinator.  Some will return one
token (the first matching item found.)  Others will return a list of all
items matching a given criterion.

These functions query the current buffer's overlay system for tokens.

@defun semantic-find-nonterminal-by-overlay &optional positionormarker buffer
Find all nonterminals covering @var{POSITIONORMARKER} by using overlays.
If @var{POSITIONORMARKER} is @code{nil}, use the current point.
Optional @var{BUFFER} is used if @var{POSITIONORMARKER} is a number, otherwise the current
buffer is used.  This finds all tokens covering the specified position
by checking for all overlays covering the current spot.  They are then sorted
from largest to smallest via the start location.
@end defun

@defun semantic-find-nonterminal-by-overlay-in-region start end &optional buffer
Find all nonterminals which exist in whole or in part between @var{START} and @var{END}.
Uses overlays to determine positin.
Optional @var{BUFFER} argument specifies the buffer to use.
@end defun

@defun semantic-current-nonterminal
Return the current nonterminal in the current buffer.
If there are more than one in the same location, return the
smallest token.
@end defun

These functions search the entire stream for tokens matches a specific
criteria.

@defun semantic-find-nonterminal-by-name name streamorbuffer &optional search-parts search-include
Find a nonterminal @var{NAME} within @var{STREAMORBUFFER}.  @var{NAME} is a string.
If @var{SEARCH-PARTS} is non-@code{nil}, search children of tokens.
If @var{SEARCH-INCLUDE} is non-@code{nil}, search include files.
@end defun

@defun semantic-find-nonterminal-by-property property value streamorbuffer &optional search-parts search-includes
Find all nonterminals with @var{PROPERTY} equal to @var{VALUE} in @var{STREAMORBUFFER}.
Properties can be added with @dfn{semantic-token-put}.
Optional argument @var{SEARCH-PARTS} and @var{SEARCH-INCLUDES} are passed to
@dfn{semantic-find-nonterminal-by-function}.
@end defun

@defun semantic-find-nonterminal-by-extra-spec spec streamorbuffer &optional search-parts search-includes
Find all nonterminals with a given @var{SPEC} in @var{STREAMORBUFFER}.
@var{SPEC} is a symbol key into the modifiers association list.
Optional argument @var{SEARCH-PARTS} and @var{SEARCH-INCLUDES} are passed to
@dfn{semantic-find-nonterminal-by-function}.
@end defun

@defun semantic-find-nonterminal-by-extra-spec-value spec value streamorbuffer &optional search-parts search-includes
Find all nonterminals with a given @var{SPEC} equal to @var{VALUE} in @var{STREAMORBUFFER}.
@var{SPEC} is a symbol key into the modifiers association list.
@var{VALUE} is the value that @var{SPEC} should match.
Optional argument @var{SEARCH-PARTS} and @var{SEARCH-INCLUDES} are passed to
@dfn{semantic-find-nonterminal-by-function}.
@end defun

@defun semantic-find-nonterminal-by-position position streamorbuffer &optional nomedian
Find a nonterminal covering @var{POSITION} within @var{STREAMORBUFFER}.
@var{POSITION} is a number, or marker.  If @var{NOMEDIAN} is non-@code{nil}, don't do
the median calculation, and return nil.
@end defun

@defun semantic-find-innermost-nonterminal-by-position position streamorbuffer &optional nomedian
Find a list of nonterminals covering @var{POSITION} within @var{STREAMORBUFFER}.
@var{POSITION} is a number, or marker.  If @var{NOMEDIAN} is non-@code{nil}, don't do
the median calculation, and return nil.
This function will find the topmost item, and recurse until no more
details are available of findable.
@end defun

@defun semantic-find-nonterminal-by-token token streamorbuffer &optional search-parts search-includes
Find all nonterminals with a token @var{TOKEN} within @var{STREAMORBUFFER}.
@var{TOKEN} is a symbol representing the type of the tokens to find.
Optional argument @var{SEARCH-PARTS} and @var{SEARCH-INCLUDE} are passed to
@dfn{semantic-find-nonterminal-by-function}.
@end defun

@defun semantic-find-nonterminal-standard streamorbuffer &optional search-parts search-includes
Find all nonterminals in @var{STREAMORBUFFER} which define simple token types.
Optional argument @var{SEARCH-PARTS} and @var{SEARCH-INCLUDE} are passed to
@dfn{semantic-find-nonterminal-by-function}.
@end defun

@defun semantic-find-nonterminal-by-type type streamorbuffer &optional search-parts search-includes
Find all nonterminals with type @var{TYPE} within @var{STREAMORBUFFER}.
@var{TYPE} is a string which is the name of the type of the token returned.
Optional argument @var{SEARCH-PARTS} and @var{SEARCH-INCLUDES} are passed to
@dfn{semantic-find-nonterminal-by-function}.
@end defun

@defun semantic-find-nonterminal-by-function function streamorbuffer &optional search-parts search-includes
Find all nonterminals in which @var{FUNCTION} match within @var{STREAMORBUFFER}.
@var{FUNCTION} must return non-@code{nil} if an element of @var{STREAM} will be included
in the new list.

If optional argument @var{SEARCH-PARTS} is non-@code{nil}, all sub-parts of tokens
are searched.  The overloadable function @dfn{semantic-nonterminal-children} is
used for the searching child lists.  If @var{SEARCH-PARTS} is the symbol
@code{'positiononly}, then only children that have positional information are
searched.

If @var{SEARCH-INCLUDES} is non-@code{nil}, then all include files are also
searched for matches.
@end defun

@defun semantic-find-nonterminal-by-function-first-match function streamorbuffer &optional search-parts search-includes
Find the first nonterminal which @var{FUNCTION} match within @var{STREAMORBUFFER}.
@var{FUNCTION} must return non-@code{nil} if an element of @var{STREAM} will be included
in the new list.
If optional argument @var{SEARCH-PARTS}, all sub-parts of tokens are searched.
The overloadable function @dfn{semantic-nonterminal-children} is used for
searching.
If @var{SEARCH-INCLUDES} is non-@code{nil}, then all include files are also
searched for matches.
@end defun

@defun semantic-recursive-find-nonterminal-by-name name buffer
Recursivly find the first occurance of @var{NAME}.
Start search with @var{BUFFER}.  Recurse through all dependencies till found.
The return item is of the form (@var{BUFFER} @var{TOKEN}) where @var{BUFFER} is the buffer
in which @var{TOKEN} (the token found to match @var{NAME}) was found.
@end defun


@node Nonterminal Completion, Override Methods, Nonterminal Streams, Utilities
@section Nonterminal completion

These functions provide ways reading the names of items in a buffer with
completion.


@defun semantic-read-symbol prompt &optional default stream filter
Read a symbol name from the user for the current buffer.
@var{PROMPT} is the prompt to use.
Optional arguments:
@var{DEFAULT} is the default choice.  If no default is given, one is read
from under point.
@var{STREAM} is the list of tokens to complete from.
@var{FILTER} is provides a filter on the types of things to complete.
@var{FILTER} must be a function to call on each element.  (See
@end defun

@defun semantic-read-variable prompt &optional default stream
Read a variable name from the user for the current buffer.
@var{PROMPT} is the prompt to use.
Optional arguments:
@var{DEFAULT} is the default choice.  If no default is given, one is read
from under point.
@var{STREAM} is the list of tokens to complete from.
@end defun

@defun semantic-read-function prompt &optional default stream
Read a function name from the user for the current buffer.
@var{PROMPT} is the prompt to use.
Optional arguments:
@var{DEFAULT} is the default choice.  If no default is given, one is read
from under point.
@var{STREAM} is the list of tokens to complete from.
@end defun

@defun semantic-read-type prompt &optional default stream
Read a type name from the user for the current buffer.
@var{PROMPT} is the prompt to use.
Optional arguments:
@var{DEFAULT} is the default choice.  If no default is given, one is read
from under point.
@var{STREAM} is the list of tokens to complete from.
@end defun


@node Override Methods, Parser Hooks, Nonterminal Completion, Utilities
@section Override Methods

These functions are called `override methods' because they provide
generic behaviors, which a given language can override.  For example,
finding a dependency file in Emacs lisp can be done with the
`locate-library' command (which overrides the default behavior.)  In C,
a dependency can be found by searching a generic search path which can
be passed in via a variable.

@menu
* Token->Text:: Converting Tokens into text strings
* Token Details:: Arbitrary token detail fetching
* Local Context:: Deriving information about a language specific local
                  context.
* Making New Methods:: How to add your own methods for a tool
@end menu

@node Token->Text, Token Details, Override Methods, Override Methods
@subsection Token->Text

Any given token consists of Meta information which is best viewed in
some textual form.  This could be as simple as the token's name, or as
a prototype to be added to header file in C.  Not only are there
several default converters from a Token into text, but there is also
some convenient variables that can be used with them.  Use these
variables to allow options on output forms when displaying tokens in
your programs.

@defvar semantic-token->text-functions
List of functions which convert a token to text.
Each function must take the parameters @var{TOKEN} &optional @var{PARENT} @var{COLOR}.
@var{TOKEN} is the token to convert.
@var{PARENT} is a parent token or name which refers to the structure
or class which contains @var{TOKEN}.  @var{PARENT} is @var{NOT} a class which a @var{TOKEN}
would claim as a parent.
@var{COLOR} indicates that the generated text should be colored using
@code{font-lock}.
@end defvar

@defvar semantic-token->text-custom-list
@var{A} List used by customizeable variables to choose a token to text function.
Use this variable in the @code{:type} field of a customizable variable.
@end defvar

Every token to text conversion function must take the same parameters,
which are @var{TOKEN}, the token to be converted, @var{PARENT}, the
containing parent (like a structure which contains a variable), and
@var{COLOR}, which is a flag specifying that color should be applied
to the returned string.

When creating, or using these strings, particularly with color, use
@dfn{concat} to build up larger strings instead of @dfn{format}.  This
will preserve text properties.

@defun semantic-name-nonterminal token &optional parent color
Return the name string describing @var{TOKEN}.
The name is the shortest possible representation.
Optional argument @var{PARENT} is the parent type if @var{TOKEN} is a detail.
Optional argument @var{COLOR} means highlight the prototype with font-lock colors.
@end defun

@defun semantic-summarize-nonterminal token &optional parent color
Summarize @var{TOKEN} in a reasonable way.
Optional argument @var{PARENT} is the parent type if @var{TOKEN} is a detail.
Optional argument @var{COLOR} means highlight the prototype with font-lock colors.
@end defun

@defun semantic-prototype-nonterminal token &optional parent color
Return a prototype for @var{TOKEN}.
This function should be overloaded, though it need not be used.
This is because it can be used to create code by language independent
tools.
Optional argument @var{PARENT} is the parent type if @var{TOKEN} is a detail.
Optional argument @var{COLOR} means highlight the prototype with font-lock colors.
@end defun

@defun semantic-prototype-file buffer
Return a file in which prototypes belonging to @var{BUFFER} should be placed.
Default behavior (if not overriden) looks for a token specifying the
prototype file, or the existence of an @var{EDE} variable indicating which
file prototypes belong in.
@end defun

@defun semantic-abbreviate-nonterminal token &optional parent color
Return an abbreviated string describing @var{TOKEN}.
The abbreviation is to be short, with possible symbols indicating
the type of token, or other information.
Optional argument @var{PARENT} is the parent type if @var{TOKEN} is a detail.
Optional argument @var{COLOR} means highlight the prototype with font-lock colors.
@end defun

@defun semantic-concise-prototype-nonterminal token &optional parent color
Return a concise prototype for @var{TOKEN}.
Optional argument @var{PARENT} is the parent type if @var{TOKEN} is a detail.
Optional argument @var{COLOR} means highlight the prototype with font-lock colors.
@end defun

@defun semantic-uml-abbreviate-nonterminal token &optional parent color
Return a @var{UML} style abbreviation for @var{TOKEN}.
Optional argument @var{PARENT} is the parent type if @var{TOKEN} is a detail.
Optional argument @var{COLOR} means highlight the prototype with font-lock colors.
@end defun

@node Token Details, Local Context, Token->Text, Override Methods
@subsection Token Details

These functions help derive information about tokens that may not
be obvious for non-traditional languages with their own token types.

@defun semantic-nonterminal-children token &optional positionalonly
Return the list of top level children belonging to @var{TOKEN}.
Children are any sub-tokens which may contain overlays.
The default behavior (if not overriden with @code{nonterminal-children}
is to return type parts for a type, and arguments for a function.

If optional argument @var{POSITIONALONLY} is non-@code{nil}, then only return valid
children if they contain positions.  Some languages may choose to create
lists of children without position/overlay information.

If this function is overriden, use @dfn{semantic-nonterminal-children-default}
to also include the default behavior, and merely extend your own.

Note for language authors:
  If a mode defines a language that has tokens in it with overlays that
should not be considered children, you should still return them with
this function.  If you do not, then token reparsing, and database
saving will fail.
@end defun

@defun semantic-nonterminal-protection token &optional parent
Return protection information about @var{TOKEN} with optional @var{PARENT}.
This function returns on of the following symbols:
   @code{nil}      - No special protection.  Language dependent.
   @code{'public}  - Anyone can access this @var{TOKEN}.
   @code{'private} - Only methods in the local scope can access @var{TOKEN}.
   @code{'friend}  - Like private, except some outer scopes are allowed
              access to token.
Some languages may choose to provide additional return symbols specific
to themselves.  Use of this function should allow for this.

The default behavior (if not overriden with @code{nonterminal-children}
is to return a symbol based on type modifiers.
@end defun

@defun semantic-find-dependency token
Find the filename represented from @var{TOKEN}.
@var{TOKEN} may be a stripped element, in which case @var{PARENT} specifies a
parent token that has positinal information.
Depends on @code{semantic-dependency-include-path} for searching.  Always searches
`.' first, then searches additional paths.


@defvar semantic-dependency-include-path
Defines the include path used when searching for files.
This should be a list of directories to search which is specific to
the file being included.
This variable can also be set to a single function.  If it is a
function, it will be called with one arguments, the file to find as a
string, and  it should return the full path to that file, or nil.
@end defvar


@end defun


@defun semantic-find-nonterminal token &optional parent
Find the location of @var{TOKEN}.
@var{TOKEN} may be a stripped element, in which case @var{PARENT} specifies a
parent token that has position information.
Different behaviors are provided depending on the type of token.
For example, dependencies (includes) will seek out the file that is
depended on, and functions will move to the specified definition.
@end defun

@defun semantic-find-documentation token
Find documentation from @var{TOKEN} and return it as a clean string.
@var{TOKEN} might have @var{DOCUMENTATION} set in it already.  If not, there may be
some documentation in a comment preceding TOKEN's definition which we
cal look for.  When appropriate, this can be overridden by a language specific
enhancement.
@end defun

@node Local Context, Making New Methods, Token Details, Override Methods
@subsection Local Context

@defun semantic-up-context &optional point
Move point up one context from @var{POINT}.
Return non-@code{nil} if there are no more context levels.
Overloaded functions using @code{up-context} take no parameters.
@end defun

@defun semantic-beginning-of-context &optional point
Move @var{POINT} to the beginning of the current context.
Return non-@code{nil} if there is no upper context.
The default behavior uses @dfn{semantic-up-context}.  It can
be overridden with @code{beginning-of-context}.
@end defun

@defun semantic-end-of-context &optional point
Move @var{POINT} to the end of the current context.
Return non-@code{nil} if there is no upper context.
Be default, this uses @dfn{semantic-up-context}, and assumes parenthetical
block delimiters.  This can be overridden with @code{end-of-context}.
@end defun

@defun semantic-get-local-variables &optional point
Get the local variables based on POINT's context.
Local variables are returned in Semantic token format.
Be default, this calculates the current bounds using context blocks
navigation, then uses the parser with @code{bovine-inner-scope} to
parse tokens at the beginning of the context.
This can be overriden with @code{get-local-variables}.
@end defun

@defun semantic-get-local-arguments &optional point
Get arguments (variables) from the current context at @var{POINT}.
Parameters are available if the point is in a function or method.
This function returns a list of tokens.  If the local token returns
just a list of strings, then this function will convert them to tokens.
Part of this behavior can be overridden with @code{get-local-arguments}.
@end defun

@defun semantic-get-all-local-variables &optional point
Get all local variables for this context, and parent contexts.
Local variables are returned in Semantic token format.
Be default, this gets local variables, and local arguments.
This can be overridden with @code{get-all-local-variables}.
Optional argument @var{POINT} is the location to start getting the variables from.
@end defun

These next set of functions handle local context parsing.   This means
looking at the code (locally) and navigating, and fetching information
such as a the type of the parameter the cursor may be typing in.

@defun semantic-end-of-command
Move to the end of the current command.
Be default, uses @code{semantic-command-separation-character}.
Override with @code{end-of-command}.
@end defun

@defun semantic-beginning-of-command
Move to the beginning of the current command.
Be default, users @code{semantic-command-separation-character}.
Override with @code{beginning-of-command}.
@end defun

@defun semantic-ctxt-current-symbol &optional point
Return the current symbol the cursor is on at @var{POINT} in a list.
This will include a list of type/field names when applicable.
This can be overridden using @code{ctxt-current-symbol}.
@end defun

@defun semantic-ctxt-current-assignment &optional point
Return the current assignment near the cursor at @var{POINT}.
Return a list as per @dfn{semantic-ctxt-current-symbol}.
Return @code{nil} if there is nothing relevant.
Override with @code{ctxt-current-assignment}.
@end defun

@defun semantic-ctxt-current-function &optional point
Return the current function the cursor is in at @var{POINT}.
The function returned is the one accepting the arguments that
the cursor is currently in.
This can be overridden with @code{ctxt-current-function}.
@end defun

@defun semantic-ctxt-current-argument &optional point
Return the current symbol the cursor is on at @var{POINT}.
Override with @code{ctxt-current-argument}.
@end defun

Local Context analysis functions can look at the local context,
and return informatin such as the type of the data that should be
entered where the cursor is.  As of this writing, these functions
need some more work.

@defun semantic-suggest-lookup-item name &optional tokentype returntype
Find a token definition matching @var{NAME} with @var{TOKENTYPE}.
Optional @var{RETURNTYPE} is a return value to match against also.
@end defun

@defun semantic-suggest-variable-token-hierarchy
Analyze the current line, and return a series of tokens.
The tokens represent a hierarchy of dereferences.  For example, a
variable name will return a list with one token representing that
variable's declaration.  If that variable is being dereferenced, then
return a list starting with the variable declaration, followed by all
fields being extracted.

For example, in c, ``foo->bar'' would return a list (@var{VARTOKEN} @var{FIELDTOKEN})
where @var{VARTOKEN} is a semantic token of the variable foo's declaration.
@var{FIELDTOKEN} is either a string, or a semantic token representing
the field in foo's type.
@end defun


@node Making New Methods, , Local Context, Override Methods
@subsection Local Context

@node Parser Hooks, , Override Methods, Utilities
@section Parser Hooks

If you write a program that uses the stream of tokens in a persistent
display or database, it is necessary to know when tokens change so
that your displays can be updated.  This is especially important as
tokens can be replaced, changed, or deleted, and the associated
overlays will then throw errors when you try to use them.  Complete
integration with token changes can be achieved via several very
important hooks.

One interesting way to interact with the parser is to let it know that
changes you are going to make will not require reparsing.

@defvar semantic-edits-are-safe
When non-@code{nil}, modifications do not require a reparse.
This prevents tokens from being marked dirty, and it
prevents top level edits from causing a cache check.
Use this when writing programs that could cause a full
reparse, but will not change the tag structure, such
as adding or updating top-level comments.
@end defvar

Next, it is sometimes useful to know what the current parsing state
is.  These function can let you know what level of reparsing may be
needed.  Careful choices on when to reparse can make your program much
faster.

@defun semantic-bovine-toplevel-full-reparse-needed-p &optional checkcache
Return non-@code{nil} if the current buffer needs a full reparse.
Optional argument @var{CHECKCACHE} indicates if the cache check should be made.
@end defun

@defun semantic-bovine-toplevel-partial-reparse-needed-p &optional checkcache
Return non-@code{nil} if the current buffer needs a partial reparse.
This only returns non-@code{nil} if @dfn{semantic-bovine-toplevel-full-reparse-needed-p}
returns nil.
Optional argument @var{CHECKCACHE} indicates if the cache check should be made
when checking @dfn{semantic-bovine-toplevel-full-reparse-needed-p}.
@end defun

If you need very close interaction with the user's editing, then these
two hooks can be used to find out when a given tag is being changed.
These hooks could even be used to cut down on reparsing if used correctly.

For all hooks, make sure you are careful to add it as a local hook if
you only want to effect a single buffer.  Setting it globally can
cause unwanted effects if your program is concerned with a single buffer.

@defvar semantic-dirty-token-hooks
Hooks run after when a token is marked as dirty (edited by the user).
The functions must take @var{TOKEN}, @var{START}, and @var{END} as a parameters.
This hook will only be called once when a token is first made dirty,
subsequent edits will not cause this to run a second time unless that
token is first cleaned.  Any token marked as dirty will
also be called with @code{semantic-clean-token-hooks}, unless a full
reprase is done instead.
@end defvar

@defvar semantic-clean-token-hooks
Hooks run after a token is marked as clean (reparsed after user edits.)
The functions must take a @var{TOKEN} as a parameter.
Any token sent to this hook will have first been called with
@code{semantic-dirty-token-hooks}.  This hook is not called for tokens
marked dirty if the buffer is completely reparsed.  In that case, use
@code{semantic-after-toplevel-bovinate-hook}.
@end defvar

@defvar semantic-change-hooks
Hooks run when semantic detects a change in a buffer.
Each hook function must take three arguments, identical to the
common hook @code{after-change-function}.
@end defvar

Lastly, if you just want to know when a buffer changes, use this hook.

@defvar semantic-after-toplevel-bovinate-hook
Hooks run after a toplevel token parse.
It is not run if the toplevel parse command is called, and buffer does
not need to be fully reparsed.
This function is also called when the toplevel cache is flushed, and
the cache is emptied.
For language specific hooks, make sure you define this as a local hook.
@end defvar

@defvar semantic-before-toplevel-cache-flush-hook
Hooks run before the toplevel nonterminal cache is flushed.
For language specific hooks, make sure you define this as a local hook.
This hook is called before a corresponding
@code{semantic-after-toplevel-bovinate-hook} which is also called during a
flush when the cache is given a new value of nil.
@end defvar

@node Current Context, Tools, Utilities, Top
@comment  node-name,  next,  previous,  up
@chapter Deriving the Current Context

This chapter deals with how to derive the current context, and also
how a language maintainter can get the current context API to work
with their language.

By default, the behavior will function in C like languages.  This
means languages with parenthetical blocks, and type dereferencing
which uses a similar form.

@menu
* Blocks::                      
* Local Variables::             Getting lists of local variables.
* Derived Context::             What goes at a given location?
* Suggestions::                 Get suggestions on one goes in a derived context.
@end menu

@node Blocks, Local Variables, Current Context, Current Context
@section Blocks and Navigation

Source code is typically built up of control structures, and blocks of
context, or lexical scope.  Semantic terms these lexical scopes as a
``context''.  The following functions can be used to navigate contexts.
Some of them are override functions.  Language authors can override
a subset of them to make them work for their language.

@defun semantic-up-context &optional point
Move point up one context from @var{POINT}.
Return non-@code{nil} if there are no more context levels.
Overloaded functions using @code{up-context} take no parameters.
@end defun

@defun semantic-beginning-of-context &optional point
Move @var{POINT} to the beginning of the current context.
Return non-@code{nil} if there is no upper context.
The default behavior uses @code{semantic-up-context}.  It can
be overridden with @code{beginning-of-context}.
@end defun

@defun semantic-end-of-context &optional point
Move @var{POINT} to the end of the current context.
Return non-@code{nil} if there is no upper context.
Be default, this uses @code{semantic-up-context}, and assumes parenthetical
block delimiters.  This can be overridden with @code{end-of-context}.
@end defun

These next set of functions can be used to navigate across commands.

@defun semantic-end-of-command
Move to the end of the current command.
Be default, uses @code{semantic-command-separation-character}.
Override with @code{end-of-command}.
@end defun

@defun semantic-beginning-of-command
Move to the beginning of the current command.
Be default, users @code{semantic-command-separation-character}.
Override with @code{beginning-of-command}.
@end defun

@node Local Variables, Derived Context, Blocks, Current Context
@section Deriving local variables

Within a given context, or block of code, local variables are often
defined.  These functions can be used to retrieve lists of locally
scoped variables.

@defun semantic-get-local-variables &optional point
Get the local variables based on POINT's context.
Local variables are returned in Semantic token format.
Be default, this calculates the current bounds using context blocks
navigation, then uses the parser with @code{bovine-inner-scope} to
parse tokens at the beginning of the context.
This can be overriden with @code{get-local-variables}.
@end defun

@defun semantic-get-local-arguments &optional point
Get arguments (variables) from the current context at @var{POINT}.
Parameters are available if the point is in a function or method.
This function returns a list of tokens.  If the local token returns
just a list of strings, then this function will convert them to tokens.
Part of this behavior can be overridden with @code{get-local-arguments}.
@end defun

@defun semantic-get-all-local-variables &optional point
Get all local variables for this context, and parent contexts.
Local variables are returned in Semantic token format.
Be default, this gets local variables, and local arguments.
This can be overridden with @code{get-all-local-variables}.
@end defun

@node Derived Context, Suggestions, Local Variables, Current Context
@section Deriving the Current Context

While a context has already been used to describe blocks of code,
other context include more local details, such as the symbol the
cursor is on, or the fact we are assinging into some other variable.

These context deriving functions can be overridden to provide language
specific behavior.  By default, it assumes a C like language.

@defun semantic-ctxt-current-symbol &optional point
Return the current symbol the cursor is on at @var{POINT} in a list.
This will include a list of type/field names when applicable.
This can be overridden using @code{ctxt-current-symbol}.
@end defun

@defun semantic-ctxt-current-assignment &optional point
Return the current assignment near the cursor at @var{POINT}.
Return a list as per @code{semantic-ctxt-current-symbol}.
Return @code{nil} if there is nothing relevant.
Override with @code{ctxt-current-assignment}.
@end defun

@defun semantic-ctxt-current-function &optional point
Return the current symbol the cursor is on at @var{POINT}.
The function returned is the one accepting the arguments that
the cursor is currently in.
This can be overridden with `ctxt.current-function'.
@end defun

@defun semantic-ctxt-current-argument &optional point
Return the current symbol the cursor is on at @var{POINT}.
Override with @code{ctxt-current-argument}.
@end defun

@node Suggestions,  , Derived Context, Current Context
@section Getting Suggestions for the Current Context

The suggestion API uses all the context deriving functions to make
intelligent guesses as to what belongs in a given location.  These
functions can be used to create smart completion mechanisms, or
displays.

Unfortunatly, they are not fully implemented in Semantic 1.4.


@node Tools, Index, Current Context, Top
@comment  node-name,  next,  previous,  up
@chapter Tools

Several tools come with Semantic which would not be possible without it.
In general, these tools will work with any language supported by
Semantic.

@menu
* speedbar::                    How to use Semantic speedbar support
* imenu::                       Special support for Imenu.
* semanticdb::                  Cache your parsed buffers between sessions.
* senator::                     The Semantic Navigator
* document::                    Document generation functions
* miscellaneous::               
@end menu

@node speedbar, imenu, Tools, Tools
@comment  node-name,  next,  previous,  up
@section Speedbar

Speedbar supports the display of tags through the Semantic parser.  To
use this utility, add a line like this to your @file{.emacs} file:

@example
(add-hook 'speedbar-load-hook (lambda () (require 'semantic-sb)))
@end example

or you can simply add:

@example
(require 'semantic-sb)
@end example

Once installed, speedbar will use semantic to find tokens, and will
display them appropriately.  Tags from semantic will have additional
details which can be seen, such as return type, or arguments to
functions.

@node imenu, semanticdb, speedbar, Tools
@comment  node-name,  next,  previous,  up
@section Imenu support

There is special support for creating Imenu entries using semantic.
This is a highly customizable tool which can create specialized menu
systems for navigating your source file.

By default, each language that wants special imenu support will set
itself up for it.  To setup imenu for your buffers, use this command
in your @file{.emacs} file:

@example
(add-hook 'semantic-init-hooks (lambda ()
                               	   (imenu-add-to-menubar "TOKENS")))
@end example

Also supported is @dfn{which-func-mode}.  This usually uses imenu
tags to show the current function.  The semantic support for this
function uses overlays, which is much faster.

You can customize imenu with the following options:

@deffn Option semantic-imenu-summary-function
Function to use when creating items in Imenu.
Some useful functions are:
@dfn{semantic-abbreviate-nonterminal}
@dfn{semantic-summarize-nonterminal}
@dfn{semantic-prototype-nonterminal}
@end deffn

@deffn Option semantic-imenu-bucketize-file
Non-@code{nil} if tokens in a file are to be grouped into buckets.
@end deffn

@deffn Option semantic-imenu-buckets-to-submenu
Non-@code{nil} if buckets of tokens are to be turned into submenus.
This option is ignored if @code{semantic-imenu-bucketize-file} is nil.
@end deffn

@deffn Option semantic-imenu-expand-type-parts
Non-@code{nil} if types should have submenus with parts in it.
@end deffn

@deffn Option semantic-imenu-bucketize-type-parts
Non-@code{nil} if elements of a type should be placed grouped into buckets.
@code{Nil} means to keep them in the same order.
Overriden to @code{nil} if @code{semantic-imenu-bucketize-file} is nil.
@end deffn

@deffn Option semantic-imenu-sort-bucket-function
Function to use when sorting tags in the buckets of functions.
@end deffn

@deffn Option semantic-imenu-index-directory
Non @code{nil} to index the entire directory for tags.
Doesn't actually parse the entire directory, but displays tags for all files
currently listed in the current Semantic database.
This variable has no meaning if semanticdb is not active.
@end deffn

@deffn Option semantic-imenu-auto-rebuild-directory-indexes
If non-@code{nil} automatically rebuild directory index imenus.
That is when a directory index imenu is updated, automatically rebuild
other buffer local ones based on the same semanticdb.
@end deffn

When adding support to a language, this variable may need to be set:

@defvar semantic-imenu-expandable-token
Tokens of this token type will be given submenu with children.
By default, a @code{type} has interesting children.  In Texinfo, however,
a @code{section} has interesting children.
@end defvar


@node semanticdb, senator, imenu, Tools
@comment  node-name,  next,  previous,  up
@section Semantic Database

Semanticdb is a utility which tracks your parsed files, and saves the
parsed information to files.  When you reload your source files,
semanticdb automatically associates the file with the cached copy,
saving time by not reparsing your buffer.

Semanticdb also provides an API for programs to use.  These functions
will return token information without loading the source file into
memory by checking the disk cache.

To use semanticdb, add the following to your @file{.emacs} file:

@example
(require 'semanticdb)
(global-semanticdb-minor-mode 1)
@end example

If you have a tool which optionally uses the semantic database, it may
be important to track if the database mode is turned on or off.

@deffn Option semanticdb-mode-hooks
Hooks run whenever @dfn{global-semanticdb-minor-mode} is run.
Use @dfn{semanticdb-minor-mode-p} to determine if the mode has been turned
on or off.
@end deffn

@deffn Option semanticdb-persistent-path
List of valid paths that semanticdb will cache tokens to.
When @dfn{global-semanticdb-minor-mode} is active, token lists will
be saved to disk when Emacs exits.  Not all directories will have
tokens that should be saved.
The value should be a list of valid paths.  @var{A} path can be a string,
indicating a directory in which to save a variable.  An element in the
list can also be a symbol.  Valid symbols are @code{never}, which will
disable any saving anywhere, @code{always}, which enables saving
everywhere, or @code{project}, which enables saving in any directory that
passes a list of predicates in @code{semantic-project-predicates}.
@end deffn

@deffn Option semanticdb-project-roots
List of directories, where each directory is the root of some project.
All subdirectories of a root project are considered a part of one project.
Values in this string can be overriden by project management programs
via the @code{semanticdb-project-root-functions} variable.
@end deffn

The important difference between these two is that you may put just
``~'' in @code{semanticdb-persistent-path}, but you may put individual
project directories into @code{semanticdb-project-roots} so that
different database lists don't get cross referenced incorrectly.

@subsection Searching

You can search for tokens in the database using the following
functions.

@defun semanticdb-find-nonterminal-by-name name &optional databases search-parts search-includes diff-mode find-file-match
Find all occurances of nonterminals with name @var{NAME} in databases.
See @dfn{semanticdb-find-nonterminal-by-function} for details on @var{DATABASES},
@var{SEARCH-PARTS}, @var{SEARCH-INCLUDES}, @var{DIFF-MODE}, and @var{FIND-@var{FILE}-MATCH}.
@end defun

@defun semanticdb-find-nonterminal-by-name-regexp regex &optional databases search-parts search-includes diff-mode find-file-match
Find all occurances of nonterminals with name matching @var{REGEX} in databases.
See @dfn{semanticdb-find-nonterminal-by-function} for details on @var{DATABASES},
@var{SEARCH-PARTS}, @var{SEARCH-INCLUDES} @var{DIFF-MODE}, and @var{FIND-@var{FILE}-MATCH}.
@end defun

@defun semanticdb-find-nonterminal-by-type type &optional databases search-parts search-includes diff-mode find-file-match
Find all nonterminals with a type of @var{TYPE} in databases.
See @dfn{semanticdb-find-nonterminal-by-function} for details on @var{DATABASES},
@var{SEARCH-PARTS}, @var{SEARCH-INCLUDES} @var{DIFF-MODE}, and @var{FIND-@var{FILE}-MATCH}.
@end defun

@defun semanticdb-find-nonterminal-by-property property value &optional databases search-parts search-includes diff-mode find-file-match
Find all nonterminals with a @var{PROPERTY} equal to @var{VALUE} in databases.
See @dfn{semanticdb-find-nonterminal-by-function} for details on @var{DATABASES},
@var{SEARCH-PARTS}, @var{SEARCH-INCLUDES} @var{DIFF-MODE}, and @var{FIND-@var{FILE}-MATCH}.
Return a list ((@var{DB-TABLE} . @var{TOKEN-LIST}) ...).
@end defun

@defun semanticdb-find-nonterminal-by-extra-spec spec &optional databases search-parts search-includes diff-mode find-file-match
Find all nonterminals with a @var{SPEC} in databases.
See @dfn{semanticdb-find-nonterminal-by-function} for details on @var{DATABASES},
@var{SEARCH-PARTS}, @var{SEARCH-INCLUDES} @var{DIFF-MODE}, and @var{FIND-@var{FILE}-MATCH}.
Return a list ((@var{DB-TABLE} . @var{TOKEN-LIST}) ...).
@end defun

@defun semanticdb-find-nonterminal-by-extra-spec-value spec value &optional databases search-parts search-includes diff-mode find-file-match
Find all nonterminals with a @var{SPEC} equal to @var{VALUE} in databases.
See @dfn{semanticdb-find-nonterminal-by-function} for details on @var{DATABASES},
@var{SEARCH-PARTS}, @var{SEARCH-INCLUDES} @var{DIFF-MODE}, and @var{FIND-@var{FILE}-MATCH}.
Return a list ((@var{DB-TABLE} . @var{TOKEN-LIST}) ...).
@end defun

@defun semanticdb-find-nonterminal-by-function function &optional databases search-parts search-includes diff-mode find-file-match
Find all occurances of nonterminals which match @var{FUNCTION}.
Search in all @var{DATABASES}.  If @var{DATABASES} is @code{nil}, search a range of
associated databases.
When @var{SEARCH-PARTS} is non-@code{nil} the search will include children of tokens.
When @var{SEARCH-INCLUDES} is non-@code{nil}, the search will include dependency files.
When @var{DIFF-MODE} is non-@code{nil}, search databases which are of a different mode.
@var{A} Mode is the @code{major-mode} that file was in when it was last parsed.
When @var{FIND-@var{FILE}-MATCH} is non-@code{nil}, the make sure any found token's file is
in an Emacs buffer.
@end defun

@defun semanticdb-file-stream file
Return a list of tokens belonging to @var{FILE}.
If file has database tokens available in the database, return them.
If file does not have tokens available, then load the file, and create them.
@end defun

@node senator, document, semanticdb, Tools
@comment  node-name,  next,  previous,  up
@section Semantic Navigator

Senator stands for SEmantic NAvigaTOR and was written by David Ponce.

This library defines commands and a minor mode to navigate between
semantic language tokens in the current buffer.

@subsection Commands

The following user level commands are provided by Senator.

@subsubsection Navigation

@deffn Command senator-next-token
Move to the next token in the current buffer.
@end deffn

@deffn Command senator-previous-token
Move to the previous token in the current buffer.
@end deffn

@deffn Command senator-jump sym
Jump to the semantic symbol @var{SYM}.
If called interactively and a prefix argument is supplied jump in the
local type's context (see function @dfn{senator-current-type-context}).
@end deffn

@subsubsection Searching

Searching using senator mode restricts the search only to the
definition text, such as the name of the functions or variables in a
given buffer.

@deffn Command senator-isearch-toggle-semantic-mode
Toggles semantic search in isearch mode.  When semantic search is
enabled, isearch is restricted to token names.
@end deffn

@deffn Command senator-search-forward string
@deffnx Command senator-search-backward string
Search forward and backward for a token matching @var{string}.
@end deffn

@deffn Command re-search-forward regex
@deffnx Command re-search-backward regex
Search forward and backward for a token matching the regular expression
@var{regex}.
@end deffn

@deffn Command word-search-forward word
@deffnx word-search-backward word
Search forward and backward for a token whose name matches @var{word}.
@end deffn

@subsubsection Completion

Completion in senator scans all known definitions in the local file,
and uses that information to provide the completion.

@deffn Command senator-complete-symbol
Complete the current symbol under point.
@end deffn

@deffn Command senator-completion-menu-keyboard-popup
Popup a completion menu for the symbol at point.
@end deffn

@subsubsection Copy/Paste

Token Copy/Paste is a high level form of the typical copy yank used by
Emacs.  Copying a token saves the meta-information related to the
function or item the cursor is currently in.  When that information is
yanked into a new buffer, the form of the text created is based on the
current status of the programming buffer.

For example, pasting a function into a different file results in a
function call template to be inserted.  In a Texinfo file, a @@deffn
is created with documentation for that function or command.

@deffn Command senator-copy-token
Take the current token, and place it in the token ring.
@end deffn

@deffn Command senator-kill-token
Take the current token, place it in the token ring, and kill it.
Killing the token removes the text for that token, and places it into
the kill ring.  Retrieve that text with @code{yank}.
@end deffn

@deffn Command senator-yank-token
Yank a token from the token ring.
The form the token takes is differnet depending on where it is being
yanked to.
@end deffn

@deffn Command senator-copy-token-to-register register &optional kill-flag
Copy the current token into @var{REGISTER}.
Optional argument @var{KILL-FLAG} will delete the text of the token to the
kill ring.
@end deffn

For programmers, to provide specialized pasting, created an override
function for @code{insert-foreign-token}. @xref{Settings}.

@subsubsection Minor Mode

@deffn Command senator-minor-mode
Toggle the SEmantic NAvigaTOR key bindings in the current buffer.

The following default key bindings are provided when semantic minor
mode is enabled:


@table @key
@item C-c , n
senator-next-token
@item C-c , p
senator-previous-token
@item C-c , i
senator-isearch-toggle-semantic-mode
@item C-c , j
senator-jump
@item C-c , TAB
senator-complete-symbol
@item C-c , SPC
senator-completion-menu-keyboard-popup
@item C-c , C-y
senator-yank-token
@item C-c , C-w
senator-kill-token
@item C-c , M-w
senator-copy-token
@item C-c , 
@end table
@end deffn

@subsection Customization

To enable the Senator keymap in all modes that support semantic parsing,
use this:

@example
(add-hook 'semantic-init-hooks 'senator-minor-mode)
@end example

To customize navigation around different types of tokens, use the
following variables:

@deffn Option senator-step-at-token-ids
List of token identifiers where to step.
Token identifier is symbol @code{'variable}, @code{'function}, @code{'type}, or other.  If
@code{nil} navigation steps at any token found.  This is a buffer local
variable.  It can be set in a mode hook to get a specific language
navigation.
@end deffn

@deffn Option senator-step-at-start-end-token-ids
List of token identifiers where to step at start and end.
Token identifier is symbol @code{'variable}, @code{'function}, @code{'type}, or other.  If
@code{nil} navigation only step at beginning of tokens.  If @code{t} step at start
and end of any token where it is allowed to step.  Also, stepping at
start and end of a token prevent stepping inside its children.  This
is a buffer local variable.  It can be set in a mode hook to get a
specific language navigation.
@end deffn

To have a mode specific customization, do something like this in a hook:

@example
(add-hook 'mode-hook
          (lambda ()
            (setq senator-step-at-token-ids '(function variable))
            (setq senator-step-at-start-end-token-ids '(function))
            ))
@end example

This will cause navigation and search commands to stop only between
functions and variables, and to step at start and end of functions
only.

@subsection Contact information for Senator

Any comments, suggestions, bug reports or upgrade requests are
welcome.  Please send them to David Ponce at david@@dponce.com

@node document, miscellaneous, senator, Tools
@comment  node-name,  next,  previous,  up
@section Document generation

The document program uses semantic token streams to aid in the
creation of texinfo documentation.  The main entry point for the
documentation generator are the following commands:

@deffn Command document &optional resetfile
Document the function or variable the cursor is in.
Optional argument @var{RESETFILE} is provided w/ universal argument.
When non-@code{nil}, query for a new documentation file.
@end deffn

@deffn Command document-inline
Document the current function with an inline comment.
@end deffn

@deffn Command document-insert-defun-comment nonterm buffer
Insert mode-comment documentation about @var{NONTERM} from @var{BUFFER}.
@end deffn

@deffn Command document-insert-new-file-header header
Insert a new header file into this buffer.  Add reference to @var{HEADER}.
Used by @code{prototype} if this file doesn't have an introductory comment.
@end deffn

In addition to these base documentation commands, the texinfo semantic
parser includes a two convenience functions when working directly with
texinfo files.

@deffn Command semantic-texi-update-doc &optional token
Update the documentation for @var{TOKEN}.
If the current buffer is a texinfo file, then find the source doc, and
update it.  If the current buffer is a source file, then get the
documentation for this item, find the existing doc in the associated
manual, and update that.
@end deffn

@deffn Command semantic-texi-goto-source &optional token
Jump to the source for the definition in the texinfo file @var{TOKEN}.
If @var{TOKEN} is @code{nil}, it is derived from the deffn under @var{POINT}.
@end deffn

@node miscellaneous,  , document, Tools
@comment  node-name,  next,  previous,  up
@section Miscellaneous commands

@deffn Command semantic-show-dirty-mode arg
Toggle highlighting of dirty tokens.  When optional argument @var{arg}
is negative, disable this mode.  A positive argument enables it.  If
@var{arg} is nil, toggle.

When active, tokens in the current buffer marked ``dirty'' are
highlighted.  A token is considered dirty if the token was generated
before the most recent change to that token's text.
@end deffn


@node Index,  , Tools, Top
@comment  node-name,  next,  previous,  up
@chapter Index

@unnumbered Function Index
@printindex fn

@bye