semantic / semantic.texi

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
1001
1002
1003
1004
1005
1006
1007
1008
1009
1010
1011
1012
1013
1014
1015
1016
1017
1018
1019
1020
1021
1022
1023
1024
1025
1026
1027
1028
1029
1030
1031
1032
1033
1034
1035
1036
1037
1038
1039
1040
1041
1042
1043
1044
1045
1046
1047
1048
1049
1050
1051
1052
1053
1054
1055
1056
1057
1058
1059
1060
1061
1062
1063
1064
1065
1066
1067
1068
1069
1070
1071
1072
1073
1074
1075
1076
1077
1078
1079
1080
1081
1082
1083
1084
1085
1086
1087
1088
1089
1090
1091
1092
1093
1094
1095
1096
1097
1098
1099
1100
1101
1102
1103
1104
1105
1106
1107
1108
1109
1110
1111
1112
1113
1114
1115
1116
1117
1118
1119
1120
1121
1122
1123
1124
1125
1126
1127
1128
1129
1130
1131
1132
1133
1134
1135
1136
1137
1138
1139
1140
1141
1142
1143
1144
1145
1146
1147
1148
1149
1150
1151
1152
1153
1154
1155
1156
1157
1158
1159
1160
1161
1162
1163
1164
1165
1166
1167
1168
1169
1170
1171
1172
1173
1174
1175
1176
1177
1178
1179
1180
1181
1182
1183
1184
1185
1186
1187
1188
1189
1190
1191
1192
1193
1194
1195
1196
1197
1198
1199
1200
1201
1202
1203
1204
1205
1206
1207
1208
1209
1210
1211
1212
1213
1214
1215
1216
1217
1218
1219
1220
1221
1222
1223
1224
1225
1226
1227
1228
1229
1230
1231
1232
1233
1234
1235
1236
1237
1238
1239
1240
1241
1242
1243
1244
1245
1246
1247
1248
1249
1250
1251
1252
1253
1254
1255
1256
1257
1258
1259
1260
1261
1262
1263
1264
1265
1266
1267
1268
1269
1270
1271
1272
1273
1274
1275
1276
1277
1278
1279
1280
1281
1282
1283
1284
1285
1286
1287
1288
1289
1290
1291
1292
1293
1294
1295
1296
1297
1298
1299
1300
1301
1302
1303
1304
1305
1306
1307
1308
1309
1310
1311
1312
1313
1314
1315
1316
1317
1318
1319
1320
1321
1322
1323
1324
1325
1326
1327
1328
1329
1330
1331
1332
1333
1334
1335
1336
1337
1338
1339
1340
1341
1342
1343
1344
1345
1346
1347
1348
1349
1350
1351
1352
1353
1354
1355
1356
1357
1358
1359
1360
1361
1362
1363
1364
1365
1366
1367
1368
1369
1370
1371
1372
1373
1374
1375
1376
1377
1378
1379
1380
1381
1382
1383
1384
1385
1386
1387
1388
1389
1390
1391
1392
1393
1394
1395
1396
1397
1398
1399
1400
1401
1402
1403
1404
1405
1406
1407
1408
1409
1410
1411
1412
1413
1414
1415
1416
1417
1418
1419
1420
1421
1422
1423
1424
1425
1426
1427
1428
1429
1430
1431
1432
1433
1434
1435
1436
1437
1438
1439
1440
1441
1442
1443
1444
1445
1446
1447
1448
1449
1450
1451
1452
1453
1454
1455
1456
1457
1458
1459
1460
1461
1462
1463
1464
1465
1466
1467
1468
1469
1470
1471
1472
1473
1474
1475
1476
1477
1478
1479
1480
1481
1482
1483
1484
1485
1486
1487
1488
1489
1490
1491
1492
1493
1494
1495
1496
1497
1498
1499
1500
1501
1502
1503
1504
1505
1506
1507
1508
1509
1510
1511
1512
1513
1514
1515
1516
1517
1518
1519
1520
1521
1522
1523
1524
1525
1526
1527
1528
1529
1530
1531
1532
1533
1534
1535
1536
1537
1538
1539
1540
1541
1542
1543
1544
1545
1546
1547
1548
1549
1550
1551
1552
1553
1554
1555
1556
1557
1558
1559
1560
1561
1562
1563
1564
1565
1566
1567
1568
1569
1570
1571
1572
1573
1574
1575
1576
1577
1578
1579
1580
1581
1582
1583
1584
1585
1586
1587
1588
1589
1590
1591
1592
1593
1594
1595
1596
1597
1598
1599
1600
1601
1602
1603
1604
1605
1606
1607
1608
1609
1610
1611
1612
1613
1614
1615
1616
1617
1618
1619
1620
1621
1622
1623
1624
1625
1626
1627
1628
1629
1630
1631
1632
1633
1634
1635
1636
1637
1638
1639
1640
1641
1642
1643
1644
1645
1646
1647
1648
1649
1650
1651
1652
1653
1654
1655
1656
1657
1658
1659
1660
1661
1662
1663
1664
1665
1666
1667
1668
1669
1670
1671
1672
1673
1674
1675
1676
1677
1678
1679
1680
1681
1682
1683
1684
1685
1686
1687
1688
1689
1690
1691
1692
1693
1694
1695
1696
1697
1698
1699
1700
1701
1702
1703
1704
1705
1706
1707
1708
1709
1710
1711
1712
1713
1714
1715
1716
1717
1718
1719
1720
1721
1722
1723
1724
1725
1726
1727
1728
1729
1730
1731
1732
1733
1734
1735
1736
1737
1738
1739
1740
1741
1742
1743
1744
1745
1746
1747
1748
1749
1750
1751
1752
1753
1754
1755
1756
1757
1758
1759
1760
1761
1762
1763
1764
1765
1766
1767
1768
1769
1770
1771
1772
1773
1774
1775
1776
1777
1778
1779
1780
1781
1782
1783
1784
1785
1786
1787
1788
1789
1790
1791
1792
1793
1794
1795
1796
1797
1798
1799
1800
1801
1802
1803
1804
1805
1806
1807
1808
1809
1810
1811
1812
1813
1814
1815
1816
1817
1818
1819
1820
1821
1822
1823
1824
1825
1826
1827
1828
1829
1830
1831
1832
1833
1834
1835
1836
1837
1838
1839
1840
1841
1842
1843
1844
1845
1846
1847
1848
1849
1850
1851
1852
1853
1854
1855
1856
1857
1858
1859
1860
1861
1862
1863
1864
1865
1866
1867
1868
1869
1870
1871
1872
1873
1874
1875
1876
1877
1878
1879
1880
1881
1882
1883
1884
1885
1886
1887
1888
1889
1890
1891
1892
1893
1894
1895
1896
1897
1898
1899
1900
1901
1902
1903
1904
1905
1906
1907
1908
1909
1910
1911
1912
1913
1914
1915
1916
1917
1918
1919
1920
1921
1922
1923
1924
1925
1926
1927
1928
1929
1930
1931
1932
1933
1934
1935
1936
1937
1938
1939
1940
1941
1942
1943
1944
1945
1946
1947
1948
1949
1950
1951
1952
1953
1954
1955
1956
1957
1958
1959
1960
1961
1962
1963
1964
1965
1966
1967
1968
1969
1970
1971
1972
1973
1974
1975
1976
1977
1978
1979
1980
1981
1982
1983
1984
1985
1986
1987
1988
1989
1990
1991
1992
1993
1994
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
2024
2025
2026
2027
2028
2029
2030
2031
2032
2033
2034
2035
2036
2037
2038
2039
2040
2041
2042
2043
2044
2045
2046
2047
2048
2049
2050
2051
2052
2053
2054
2055
2056
2057
2058
2059
2060
2061
2062
2063
2064
2065
2066
2067
2068
2069
2070
2071
2072
2073
2074
2075
2076
2077
2078
2079
2080
2081
2082
2083
2084
2085
2086
2087
2088
2089
2090
2091
2092
2093
2094
2095
2096
2097
2098
2099
2100
2101
2102
2103
2104
2105
2106
2107
2108
2109
2110
2111
2112
2113
2114
2115
2116
2117
2118
2119
2120
2121
2122
2123
2124
2125
2126
2127
2128
2129
2130
2131
2132
2133
2134
2135
2136
2137
2138
2139
2140
2141
2142
2143
2144
2145
2146
2147
2148
2149
2150
2151
2152
2153
2154
2155
2156
2157
2158
2159
2160
2161
2162
2163
2164
2165
2166
2167
2168
2169
2170
2171
2172
2173
2174
2175
2176
2177
2178
2179
2180
2181
2182
2183
2184
2185
2186
2187
2188
2189
2190
2191
2192
2193
2194
2195
2196
2197
2198
2199
2200
2201
2202
2203
2204
2205
2206
2207
2208
2209
2210
2211
2212
2213
2214
2215
2216
2217
2218
2219
2220
2221
2222
2223
2224
2225
2226
2227
2228
2229
2230
2231
2232
2233
2234
2235
2236
2237
2238
2239
2240
2241
2242
2243
2244
2245
2246
2247
2248
2249
2250
2251
2252
2253
2254
2255
2256
2257
2258
2259
2260
2261
2262
2263
2264
2265
2266
2267
2268
2269
2270
2271
2272
2273
2274
2275
2276
2277
2278
2279
2280
2281
2282
2283
2284
2285
2286
2287
2288
2289
2290
2291
2292
2293
2294
2295
2296
2297
2298
2299
2300
2301
2302
2303
2304
2305
2306
2307
2308
2309
2310
2311
2312
2313
2314
2315
2316
2317
2318
2319
2320
2321
2322
2323
2324
2325
2326
2327
2328
2329
2330
2331
2332
2333
2334
2335
2336
2337
2338
2339
2340
2341
2342
2343
2344
2345
2346
2347
2348
2349
2350
2351
2352
2353
2354
2355
2356
2357
2358
2359
2360
2361
2362
2363
2364
2365
2366
2367
2368
2369
2370
2371
2372
2373
2374
2375
2376
2377
2378
2379
2380
2381
2382
2383
2384
2385
2386
2387
2388
2389
2390
2391
2392
2393
2394
2395
2396
2397
2398
2399
2400
2401
2402
2403
2404
2405
2406
2407
2408
2409
2410
2411
2412
2413
2414
2415
2416
2417
2418
2419
2420
2421
2422
2423
2424
2425
2426
2427
2428
2429
2430
2431
2432
2433
2434
2435
2436
2437
2438
2439
2440
2441
2442
2443
2444
2445
2446
2447
2448
2449
2450
2451
2452
2453
2454
2455
2456
2457
2458
2459
2460
2461
2462
2463
2464
2465
2466
2467
2468
2469
2470
2471
2472
2473
2474
2475
2476
2477
2478
2479
2480
2481
2482
2483
2484
2485
2486
2487
2488
2489
2490
2491
2492
2493
2494
2495
2496
2497
2498
2499
2500
2501
2502
2503
2504
2505
2506
2507
2508
2509
2510
2511
2512
2513
2514
2515
2516
2517
2518
2519
2520
2521
2522
2523
2524
2525
2526
2527
2528
2529
2530
2531
2532
2533
2534
2535
2536
2537
2538
2539
2540
2541
2542
2543
2544
2545
2546
2547
2548
2549
2550
2551
2552
2553
2554
2555
2556
2557
2558
2559
2560
2561
2562
2563
2564
2565
2566
2567
2568
2569
2570
2571
2572
2573
2574
2575
2576
2577
2578
2579
2580
2581
2582
2583
2584
2585
2586
2587
2588
2589
2590
2591
2592
2593
2594
2595
2596
2597
2598
2599
2600
2601
2602
2603
2604
2605
2606
2607
2608
2609
2610
2611
2612
2613
2614
2615
2616
2617
2618
2619
2620
2621
2622
2623
2624
2625
2626
2627
2628
2629
2630
2631
2632
2633
2634
2635
2636
2637
2638
2639
2640
2641
2642
2643
2644
2645
2646
2647
2648
2649
2650
2651
2652
2653
2654
2655
2656
2657
2658
2659
2660
2661
2662
2663
2664
2665
2666
2667
2668
2669
2670
2671
2672
2673
2674
2675
2676
2677
2678
2679
2680
2681
2682
2683
2684
2685
2686
2687
2688
2689
2690
2691
2692
2693
2694
2695
2696
2697
2698
2699
2700
2701
2702
2703
2704
2705
2706
2707
2708
2709
2710
2711
2712
2713
2714
2715
2716
2717
2718
2719
2720
2721
2722
2723
2724
2725
2726
2727
2728
2729
2730
2731
2732
2733
2734
2735
2736
2737
2738
2739
2740
2741
2742
2743
2744
2745
2746
2747
2748
2749
2750
2751
2752
2753
2754
2755
2756
2757
2758
2759
2760
2761
2762
2763
2764
2765
2766
2767
2768
2769
2770
2771
2772
2773
2774
2775
2776
2777
2778
2779
2780
2781
2782
2783
2784
2785
2786
2787
2788
2789
2790
2791
2792
2793
2794
2795
2796
2797
2798
2799
2800
2801
2802
2803
2804
2805
2806
2807
2808
2809
2810
2811
2812
2813
2814
2815
2816
2817
2818
2819
2820
2821
2822
2823
2824
2825
2826
2827
2828
2829
2830
2831
2832
2833
2834
2835
2836
2837
2838
2839
2840
2841
2842
2843
2844
2845
2846
2847
2848
2849
2850
2851
2852
2853
2854
2855
2856
2857
2858
2859
2860
2861
2862
2863
2864
2865
2866
2867
2868
2869
2870
2871
2872
2873
2874
2875
2876
2877
2878
2879
2880
2881
2882
2883
2884
2885
2886
2887
2888
2889
2890
2891
2892
2893
2894
2895
2896
2897
2898
2899
2900
2901
2902
2903
2904
2905
2906
2907
2908
2909
2910
2911
2912
2913
2914
2915
2916
2917
2918
2919
2920
2921
2922
2923
2924
2925
2926
2927
2928
2929
2930
2931
2932
2933
2934
2935
2936
2937
2938
2939
2940
2941
2942
2943
2944
2945
2946
2947
2948
2949
2950
2951
2952
2953
2954
2955
2956
2957
2958
2959
2960
2961
2962
2963
2964
2965
2966
2967
2968
2969
2970
2971
2972
2973
2974
2975
2976
2977
2978
2979
2980
2981
2982
2983
2984
2985
2986
2987
2988
2989
2990
2991
2992
2993
2994
2995
2996
2997
2998
2999
3000
3001
3002
3003
3004
3005
3006
3007
3008
3009
3010
3011
3012
3013
3014
3015
3016
3017
3018
3019
3020
3021
3022
3023
3024
3025
3026
3027
3028
3029
3030
3031
3032
3033
3034
3035
3036
3037
3038
3039
3040
3041
3042
3043
3044
3045
3046
3047
3048
3049
3050
3051
3052
3053
3054
3055
3056
3057
3058
3059
3060
3061
3062
3063
3064
3065
3066
3067
3068
3069
3070
3071
3072
3073
3074
3075
3076
3077
3078
3079
3080
3081
3082
3083
3084
3085
3086
3087
3088
3089
3090
3091
3092
3093
3094
3095
3096
3097
3098
3099
3100
3101
3102
3103
3104
3105
3106
3107
3108
3109
3110
3111
3112
3113
3114
3115
3116
3117
3118
3119
3120
3121
3122
3123
3124
3125
3126
3127
3128
3129
3130
3131
3132
3133
3134
3135
3136
3137
3138
3139
3140
3141
3142
3143
3144
3145
3146
3147
3148
3149
3150
3151
3152
3153
3154
3155
3156
3157
3158
3159
3160
3161
3162
3163
3164
3165
3166
3167
3168
3169
3170
3171
3172
3173
3174
3175
3176
3177
3178
3179
3180
3181
3182
3183
3184
3185
3186
3187
3188
3189
3190
3191
3192
3193
3194
3195
3196
3197
3198
3199
3200
3201
3202
3203
3204
3205
3206
3207
3208
3209
3210
3211
3212
3213
3214
3215
3216
3217
3218
3219
3220
3221
3222
3223
3224
3225
3226
3227
3228
3229
3230
3231
3232
3233
3234
3235
3236
3237
3238
3239
3240
3241
3242
3243
3244
3245
3246
3247
3248
3249
3250
3251
3252
3253
3254
3255
3256
3257
3258
3259
3260
3261
3262
3263
3264
3265
3266
3267
3268
3269
3270
3271
3272
3273
3274
3275
3276
3277
3278
3279
3280
3281
3282
3283
3284
3285
3286
3287
3288
3289
3290
3291
3292
3293
3294
3295
3296
3297
3298
3299
3300
3301
3302
3303
3304
3305
3306
3307
3308
3309
3310
3311
3312
3313
3314
3315
3316
3317
3318
3319
3320
3321
3322
3323
3324
3325
3326
3327
3328
3329
3330
3331
3332
3333
3334
3335
3336
3337
3338
3339
3340
3341
3342
3343
3344
3345
3346
3347
3348
3349
3350
3351
3352
3353
3354
3355
3356
3357
3358
3359
3360
3361
3362
3363
3364
3365
3366
3367
3368
3369
3370
3371
3372
3373
3374
3375
3376
3377
3378
3379
3380
3381
3382
3383
3384
3385
3386
3387
3388
3389
3390
3391
3392
3393
3394
3395
3396
3397
3398
3399
3400
3401
3402
3403
3404
3405
3406
3407
3408
3409
3410
3411
3412
3413
3414
3415
3416
3417
3418
3419
3420
3421
3422
3423
3424
3425
3426
3427
3428
3429
3430
3431
3432
3433
3434
3435
3436
3437
3438
3439
3440
3441
3442
3443
3444
3445
3446
3447
3448
3449
3450
3451
3452
3453
3454
3455
3456
3457
3458
3459
3460
3461
3462
3463
3464
3465
3466
3467
3468
3469
3470
3471
3472
3473
3474
3475
3476
3477
3478
3479
3480
3481
3482
3483
3484
3485
3486
3487
3488
3489
3490
3491
3492
3493
3494
3495
3496
3497
3498
3499
3500
3501
3502
3503
3504
3505
3506
3507
3508
3509
3510
3511
3512
3513
3514
3515
3516
3517
3518
3519
3520
3521
3522
3523
\input texinfo   @c -*-texinfo-*-
@c
@c $Id$
@c
@setfilename semantic.info
@settitle Semantic Bovinator: Parser generation for Emacs

@ifinfo
@format
START-INFO-DIR-ENTRY
* semantic: (semantic).       Semantic Parsing for Emacs
END-INFO-DIR-ENTRY
@end format
@end ifinfo

@titlepage
@sp 10
@center @titlefont{Semantic}
@vskip 0pt plus 1 fill
Copyright @copyright{} 1999, 2000, 2001, 2002 Eric M. Ludlam
@end titlepage

@node Top, Install, (dir), (dir)
@comment  node-name,  next,  previous,  up

Semantic is a program for Emacs which includes, at its core, a lexer,
and a compiler compiler (bovinator).  Additional tools include a
bnf->semantic table converter, example tables, and a speedbar tool.

The core utility is the ``semantic bovinator'' which has similar
behaviors as yacc or bison.  Since it is not designed to be as feature
rich as these tools, it uses the term ``bovine'' for cow, a lesser
cousin of the yak and bison.

To send bug reports, or participate in discussions about semantic,
use the mailing list cedet-semantic@@sourceforge.net via the URL:
@url{http://lists.sourceforge.net/lists/listinfo/cedet-semantic}

@menu
* Install::                     Installing semantic.
* Overview::                    Introduce basic concepts.
* Semantic Components::         Enumerate all semantic modules.
* Lexing::                      Setting up the lexer for your language.
* Bovinating::                  Setting up the parser for your language.
* BNF conversion::              Using the BNF converter to make tables.
* Compiling::                   Running the bovinator on a source file.
* Debugging::                   Debugging bovine tables
* Programming::                 How to program with a nonterminal stream.
* Current Context::             How to get the current code context.
* Tools::                       User Tools which use semantic.
* Index::
@end menu

@node Install, Overview, Top, Top
@chapter Installation

To install semantic, untar the distribution into a subdirectory, such as
@file{/usr/share/emacs/site-lisp/semantic-#.#}.  Next, add the following
lines into your individual @file{.emacs} file, or into
@file{site-lisp/site-start.el}.

@example
(setq semantic-load-turn-everything-on t)
(load-file "/path/to/semantic/semantic-load.el")
@end example

If you would like to turn individual tools on or off in your init
file, skip the first line.

@node Overview, Semantic Components, Install, Top
@chapter Overview

Semantic is a tool primarily for the Emacs-Lisp programmer.
However, it comes with ``applications'' that non-programmer might
find useful.
This chapter is mostly for the benefit of these non-programmers
as it gives brief descriptions of basic concepts such as
grammars, parsers, compiler-compilers, parse-tree, etc.

@cindex grammar
The grammar of a natural language defines rules by which valid phrases
and sentences can be composed using words, the fundamental units with
which all sentences are created.
@cindex context-free grammar
In a similar fashion, a ``context-free grammar'' defines the rules by which
programs can be composed using the fundamental units of the language,
i.e., numbers, symbols, punctuations, etc.
Context-free grammars are often specified in a well-known form called
@cindex Backus-Naur Form
@cindex BNF
Backus-Naur Form, BNF for short.
This is a systematic way of representing context-free grammars
such that programs can read files with grammars written in BNF
and generate code for ``parser'' of that language.
@cindex yacc
@cindex compiler-compiler
YACC (Yet Another Compiler Compiler) is one such program that has been
part of UNIX operating systems since the 1970's.
YACC is pronounced the same as ``yak'', the long-haired ox found in Asia.
The parser generated by YACC is usually a C program.
@cindex bison
@uref{http://www.gnu.org/software/bison/bison.html , Bison}
is also a ``compiler compiler'' that takes BNF grammars and produces
parsers in C language.
The difference between YACC and Bison is that Bison is
@cindex free software
@uref{http://www.gnu.org/philosophy/free-sw.html , free software}
and upward-compatible with YACC.
It also comes with an excellent manual.

Semantic is similar in spirit to YACC and Bison.
@cindex bovinator
Semantic, however, is referred to as a @dfn{bovinator} rather than
as a parser, because it is a lesser cousin of YACC and Bison.
It is lesser in that it does not perform a full parse
like YACC or Bison.
@cindex bovination
Instead, it @dfn{bovinates}.
``Bovination'' refers to partial parsing which
@cindex parse tree
creates @dfn{parse trees} of only the top most
expressions rather than parsing every nested expression.
This is sufficient for the purposes for which semantic was designed.
Semantic is meant to be used within Emacs for providing
editor-related features such as code browsers and translators rather
than for compiling which requires far more complex and complete parsers.
Semantic is not designed to be able to create full parse trees.

@cindex parser
One key benefit of semantic is that it creates parse trees
@cindex bovine tree
(perhaps the term @dfn{bovine tree} may be more accurate)
with the same structure regardless of the type of language involved.
Higher level applications written to work with bovine trees
will then work with any language for which the grammar is available.
For example, a code browser written today that supports C, C++, and
Java may work without any change on other languages that do not even
exist yet.
All one has to do is to write the BNF specification for the new language.
The rest of the work is done by semantic.
For certain languages, it is hard if not impossible to specify the syntax
of the language in BNF form, e.g.,
@uref{http://www.texinfo.org ,texinfo}
and other document oriented languages.
Semantic provides a parser for texinfo nevertheless.
Instead of BNF grammar, texinfo files are ``parsed'' using
@ref{Regexps,regular-expressions,regular-expressions,emacs}.

Semantic comes with grammars for these languages:

@itemize @bullet
@item C
@item Emacs-Lisp
@item java
@item makefile
@item scheme
@end itemize

Several tools employing semantic that provide user observable features
are listed in @ref{Tools} section.

@node Semantic Components, Lexing, Overview, Top
@chapter Semantic Components

This chapter gives an overview of major components
of semantic and how they interact with each other to perform its job.

The first step of parsing is to break up the input file into
its fundamental components.
This step is called lexing.
The output of the lexer is a list of tokens that make up the file.

@example
        syntax table, keywords list, and options
                         |
                         |
                         v
    input file  ---->  Lexer   ----> token stream
@end example

The next step is the parsing shown below.

@example
                    bovine table
                         |
                         v
    token stream --->  Parser  ----> parse tree
@end example

The end result, the parse tree, is created based on the ``bovine table'',
which is the internal representation of the BNF language grammar used by
semantic.

Semantic database provides caching of the parse trees by saving them
into files named @file{semantic.cache} automatically then loading them
when appropriate instead of re-parsing.  The reason for this is to save the
time it takes to parse a file which could take several seconds or more
for large files.

Finally, semantic provides an API for the Emacs-Lisp programmer to access the
information in the parse tree.

@node Lexing, Bovinating, Semantic Components, Top
@chapter Preparing your language for Lexing

In order to reduce a source file into a token list, it must first be
converted into a token stream.  Tokens are syntactic elements such as
whitespace, symbols, strings, lists, and punctuation.

The lexer uses the major-mode's syntax table for conversion.
@xref{Syntax Tables,,,elisp}.
As long as that is set up correctly (along with the important
@code{comment-start} and @code{comment-start-skip} variable) the lexer
should already work for your language.

The primary entry point of the lexer is the @dfn{semantic-flex} function
shown below.
Normally, you do not need to call this function.
It is usually called by @emph{semantic-bovinate-toplevel} for you.

@anchor{semantic-flex}
@defun semantic-flex start end &optional depth length
Using the syntax table, do something roughly equivalent to flex.
Semantically check between @var{START} and @var{END}.  Optional
argument @var{DEPTH} indicates at what level to scan over entire
lists.  The return value is a token stream.  Each element is a list
of the form (symbol start-expression .  end-expresssion). @var{END} does
not mark the end of the text scanned, only the end of the beginning of
text scanned.  Thus, if a string extends past @var{END}, the end of
the return token will be larger than @var{END}.  To truly restrict
scanning, use `narrow-to-region'.  The last argument, @var{LENGTH}
specifies that @dfn{semantic-flex} should only return
@var{LENGTH} tokens.
@end defun

@menu
* Lexer Overview::
* Lexer Output::
* Lexer Options::
* Keywords::
* Keyword Properties::
@end menu

@node Lexer Overview, Lexer Output, Lexing, Lexing
@section Lexer Overview

Semantic lexer breaks up the content of an Emacs buffer into a list of
tokens.  This process is based mostly on regular expressions which in
turn depend on the syntax table of the buffer's major mode being setup
properly.
@xref{Major Modes,,,emacs}.
@xref{Syntax Tables,,,elisp}.
@xref{Regexps,,,emacs}.

Specifically, the following regular expressions which rely on syntax
tables are used:

@table @code
@item @code{\\s-}
whitespace characters
@item @code{\\sw}
word constituent
@item @code{\\s_}
symbol constituent
@item @code{\\s.}
punctuation character
@item @code{\\s<}
comment starter
@item @code{\\s>}
comment ender
@item @code{\\s\\}
escape character
@item @code{\\s)}
close parenthesis character
@item @code{\\s$}
paired delimiter
@item @code{\\s\"}
string quote
@item @code{\\s\'}
expression prefix
@end table

In addition, Emacs' built-in features such as
@code{comment-start-skip},
@code{forward-comment},
@code{forward-list},
and
@code{forward-sexp}
are employed.

@node Lexer Output, Lexer Options, Lexer Overview, Lexing
@section Lexer Output

The lexer, @ref{semantic-flex}, scans the content of a buffer and
returns a token list.
Let's illustrate this using this simple example.

@example
00: /*
01:  * Simple program to demonstrate semantic.
02:  */
03:
04: #include <stdio.h>
05:
06: int i_1;
07:
08: int
09: main(int argc, char** argv)
10: @{
11:     printf("Hello world.\n");
12: @}
@end example

Evaluating @code{(semantic-flex (point-min) (point-max))}
within the buffer with the code above returns the following token list.
The input line and string that produced each token is shown after
each semi-colon.

@example
((punctuation     52 .  53)     ; 04: #
 (INCLUDE         53 .  60)     ; 04: include
 (punctuation     61 .  62)     ; 04: <
 (symbol          62 .  67)     ; 04: stdio
 (punctuation     67 .  68)     ; 04: .
 (symbol          68 .  69)     ; 04: h
 (punctuation     69 .  70)     ; 04: >
 (INT             72 .  75)     ; 06: int
 (symbol          76 .  79)     ; 06: i_1
 (punctuation     79 .  80)     ; 06: ;
 (INT             82 .  85)     ; 08: int
 (symbol          86 .  90)     ; 08: main
 (semantic-list   90 . 113)     ; 08: (int argc, char** argv)
 (semantic-list  114 . 147)     ; 09-12: body of main function
 )
@end example

As shown above, the token list is a list of ``tokens''.
Each token in turn is a list of the form

@example
(TOKEN-TYPE BEGINNING-POSITION . ENDING-POSITION)
@end example

@noindent
where TOKEN-TYPE is a symbol, and the other two are integers indicating
the buffer position that delimit the token such that

@lisp
(buffer-substring BEGINNING-POSITION ENDING-POSITION)
@end lisp

@noindent
would return the string form of the token.

Note that one line (line 4 above) can produce seven tokens while
the whole body of the function produces a single token.
This is because the @var{depth} parameter of @code{semantic-flex} was
not specified.
Let's see the output when @var{depth} is set to 1.
Evaluate @code{(semantic-flex (point-min) (point-max) 1)} in the same buffer.
Note the third argument of @code{1}.

@example
((punctuation    52 .  53)     ; 04: #
 (INCLUDE        53 .  60)     ; 04: include
 (punctuation    61 .  62)     ; 04: <
 (symbol         62 .  67)     ; 04: stdio
 (punctuation    67 .  68)     ; 04: .
 (symbol         68 .  69)     ; 04: h
 (punctuation    69 .  70)     ; 04: >
 (INT            72 .  75)     ; 06: int
 (symbol         76 .  79)     ; 06: i_1
 (punctuation    79 .  80)     ; 06: ;
 (INT            82 .  85)     ; 08: int
 (symbol         86 .  90)     ; 08: main

 (open-paren     90 .  91)     ; 08: (
 (INT            91 .  94)     ; 08: int
 (symbol         95 .  99)     ; 08: argc
 (punctuation    99 . 100)     ; 08: ,
 (CHAR          101 . 105)     ; 08: char
 (punctuation   105 . 106)     ; 08: *
 (punctuation   106 . 107)     ; 08: *
 (symbol        108 . 112)     ; 08: argv
 (close-paren   112 . 113)     ; 08: )

 (open-paren    114 . 115)     ; 10: @{
 (symbol        120 . 126)     ; 11: printf
 (semantic-list 126 . 144)     ; 11: ("Hello world.\n")
 (punctuation   144 . 145)     ; 11: ;
 (close-paren   146 . 147)     ; 12: @}
 )
@end example

The @var{depth} parameter ``peeled away'' one more level of ``list''
delimited by matching parenthesis or braces.
The depth parameter can be specified to be any number.
However, the parser needs to be able to handle the extra tokens.

This is an interesting benefit of the lexer having the full
resources of Emacs at its disposal.
Skipping over matched parenthesis is achieved by simply calling
the built-in functions @code{forward-list} and @code{forward-sexp}.

All common token symbols are enumerated below.  Additional token
symbols aside from these can be generated by the lexer if user option
@var{semantic-flex-extensions} is set.  It is up to the user to add
matching extensions to the parser to deal with the lexer
extensions.  An example use of @var{semantic-flex-extensions} is in
@file{semantic-make.el} where @var{semantic-flex-extensions} is set to
the value of @var{semantic-flex-make-extensions} which may generate
@code{shell-command} tokens.

@anchor{Default syntactic tokens}
@subsection Default syntactic tokens if the lexer is not extended.
@table @code
@item bol
Empty string matching a beginning of line.
This token is produced only if the user set
@var{semantic-flex-enable-bol} to non-@code{nil}.
@item charquote
String sequences that match @code{\\s\\+}.
@item close-paren
Characters that match @code{\\s)}.
These are typically @code{)}, @code{@}}, @code{]}, etc.
@item comment
A comment chunk.
These token types are not produced by default.
They are produced only if the user set
@var{semantic-ignore-comments} to @code{nil}.
@item newline
Characters matching @code{\\s-*\\(\n\\|\\s>\\)}.
This token is produced only if the user set
@var{semantic-flex-enable-newlines} to
non-@code{nil}.
@item open-paren
Characters that match @code{\\s(}.
These are typically @code{(}, @code{@{}, @code{[}, etc.
Note that these are not usually generated unless the @var{depth}
argument to @ref{semantic-flex} is greater than 0.
@item punctuation
Characters matching @code{\\(\\s.\\|\\s$\\|\\s'\\)}.
@item semantic-list
String delimited by matching parenthesis, braces, etc. that
the lexer skipped over, because the @var{depth} parameter
to @ref{semantic-flex} was not high enough.
@item string
Quoted strings, i.e., string sequences that start and end with
characters matching @code{\\s\"}.
The lexer relies on @code{forward-sexp} to find the
matching end.
@item symbol
String sequences that match @code{\\(\\sw\\|\\s_\\)+}.
@item whitespace
Characters that match `\\s-+' regexp. 
This token is produced only if the user set
@var{semantic-flex-enable-whitespace} to non-@code{nil}.  If
@var{semantic-ignore-comments} is non-@code{nil} too comments are
considered as whitespaces.
@end table

@node Lexer Options, Keywords, Lexer Output, Lexing
@section Lexer Options

Although most lexer functions are called for you by other semantic
functions, there are ways for you to extend or customize the lexer.
Three variables shown below serve this purpose.

@defvar semantic-flex-unterminated-syntax-end-function
Function called when unterminated syntax is encountered.
This should be set to one function.  That function should take three
parameters.  The @var{SYNTAX}, or type of syntax which is unterminated.
@var{SYNTAX-START} where the broken syntax begins.
@var{FLEX-END} is where the lexical analysis was asked to end.
This function can be used for languages that can intelligently fix up
broken syntax, or the exit lexical analysis via @dfn{throw} or @dfn{signal}
when finding unterminated syntax.
@end defvar

@defvar semantic-flex-extensions
Buffer local extensions to the lexical analyzer.
This should contain an alist with a key of a regex and a data element of
a function.  The function should both move point, and return a lexical
token of the form:

@example
( TYPE START . END)
@end example

@code{nil} is a valid return value.
@var{TYPE} can be any type of symbol, as long as it doesn't occur as a
nonterminal in the language definition.
@end defvar

@defvar semantic-flex-syntax-modifications
Changes the syntax table for a given buffer.
These changes are active only while the buffer is being flexed.
This is a list where each element has the form
@example
(CHAR CLASS)
@end example

CHAR is the char passed to `modify-syntax-entry',
and CLASS is the string also passed to `modify-syntax-entry' to define
what syntax class CHAR has.

@example
(setq semantic-flex-syntax-modifications '((?. "_"))
@end example

This makes the period @asis{.} a symbol constituent.  This may be necessary if
filenames are prevalent, such as in Makefiles.
@end defvar

@defvar semantic-flex-enable-newlines
When flexing, report @code{'newlines} as syntactic elements.
Useful for languages where the newline is a special case terminator.
Only set this on a per mode basis, not globally.
@end defvar

@defvar semantic-flex-enable-whitespace
When flexing, report @code{'whitespace} as syntactic elements.
Useful for languages where the syntax is whitespace dependent.
Only set this on a per mode basis, not globally.
@end defvar

@defvar semantic-flex-enable-bol
When flexing, report beginning of lines as syntactic elements.
Useful for languages like python which are indentation sensitive.
Only set this on a per mode basis, not globally.
@end defvar

@defvar semantic-number-expression
Regular expression for matching a number.
If this value is @code{nil}, no number extraction is done during lex.
This expression tries to match C and Java like numbers.

@example
DECIMAL_LITERAL:
    [1-9][0-9]*
  ;
HEX_LITERAL:
    0[xX][0-9a-fA-F]+
  ;
OCTAL_LITERAL:
    0[0-7]*
  ;
INTEGER_LITERAL:
    <DECIMAL_LITERAL>[lL]?
  | <HEX_LITERAL>[lL]?
  | <OCTAL_LITERAL>[lL]?
  ;
EXPONENT:
    [eE][+-]?[09]+
  ;
FLOATING_POINT_LITERAL:
    [0-9]+[.][0-9]*<EXPONENT>?[fFdD]?
  | [.][0-9]+<EXPONENT>?[fFdD]?
  | [0-9]+<EXPONENT>[fFdD]?
  | [0-9]+<EXPONENT>?[fFdD]
  ;
@end example
@end defvar

@node Keywords, Keyword Properties, Lexer Options, Lexing
@section Keywords

Another important piece of the lexer is the keyword table (see
@ref{Settings}).  You language will want to set up a keyword table for
fast conversion of symbol strings to language terminals.

The keywords table can also be used to store additional information
about those keywords.  The following programming functions can be useful
when examining text in a language buffer.

@defun semantic-flex-keyword-p text
Return non-@code{nil} if @var{TEXT} is a keyword in the keyword table.
@end defun

@defun semantic-flex-keyword-put text property value
For keyword @var{TEXT}, set @var{PROPERTY} to @var{VALUE}.
@end defun

@defun semantic-token-put-no-side-effect token key value
For @var{TOKEN}, put the property @var{KEY} on it with @var{VALUE} without side effects.
If @var{VALUE} is @code{nil}, then remove the property from @var{TOKEN}.
All cons cells in the property list are replicated so that there
are no side effects if @var{TOKEN} is in shared lists.
@end defun

@defun semantic-flex-keyword-get text property
For keyword @var{TEXT}, get the value of @var{PROPERTY}.
@end defun

@defun semantic-flex-map-keywords fun &optional property
Call function @var{FUN} on every semantic keyword.
If optional @var{PROPERTY} is non-nil, call @var{FUN} only on every
keyword which has a @var{PROPERTY} value.
@var{FUN} receives a semantic keyword as argument.
@end defun

@defun semantic-flex-keywords &optional property
Return a list of semantic keywords.
If optional @var{PROPERTY} is non-nil, return only keywords which have
@var{PROPERTY} set.
@end defun

Keyword properties can be set up in a BNF file for ease of maintenance.
While examining the text in a language buffer, this can provide an easy
and quick way of storing details about text in the buffer.

@node Keyword Properties,  , Keywords, Lexing
@section Standard Keyword Properties

Add known properties here when they are known.

@node Bovinating, BNF conversion, Lexing, Top
@chapter Preparing a bovine table for your language

When converting a source file into a nonterminal token stream
(parse-tree) it is important to specify rules to accomplish this.  The
rules are stored in the buffer local variable
@code{semantic-toplevel-bovine-table}.

While it is certainly possible to write this table yourself, it is most
likely you will want to use the BNF converter (see @xref{BNF conversion}.)
This is an easier method for specifying your rules.  You will still need
to specify a variable in your language for the table, however.  A good
rule of thumb is to call it @code{language-toplevel-bovine-table} if it
part of the language, or @code{semantic-toplevel-language-bovine-table}
if you donate it to the semantic package.

When initializing a major-mode for your language, you will set the
variable @code{semantic-toplevel-bovine-table} to the contents of your
language table.  @code{semantic-toplevel-bovine-table} is always buffer
local.

Since it is important to know the format of the table when debugging ,
you should still attempt to understand the basics of the table.

Please see the documentation for the variable
@code{semantic-toplevel-bovine-table} for details on its format.

* add more doc here *

@node BNF conversion, Compiling, Bovinating, Top
@chapter Using the BNF converter to make bovine tables

The BNF converter takes a file in "Bovine Normal Form" which is similar
to "Backus-Naur Form".  If you have ever used yacc or bison, you will
find it similar.  The BNF form used by semantic, however, does not
include token precedence rules, and several other features needed to make
real parser generators.

It is important to have an Emacs Lisp file with a variable ready to take
the output of your table (see @xref{Bovinating}.)  Also, make sure that the
file @file{semantic-bnf.el} is loaded.  Give your language file the
extension @file{.bnf} and you are ready.

The comment character is @asis{#}.

When you want to test your file, use the keyboard shortcut @kbd{C-c C-c}
to parse the file, generate the variable, and load the new definition
in.  It will then use the settings specified above to determine what to
do.  Use the shortcut @kbd{C-c c} to do the same thing, but spend
extra time indenting the table nicely.

Make sure that you create the variable specified in the
@code{%parsetable} token before trying to convert the BNF file.  A
simple definition like this is sufficient.

@example
(defvar semantic-toplevel-lang-bovine-table
   nil
   "Table for use with semantic for parsing LANG.")
@end example

If you use tokens (created with the @code{%token} specifier), also
make sure you have a keyword table available, like this:

@example
(defvar semantic-lang-keyword-table
   nil
   "Table for use with semantic for keywords.")
@end example

Specify the name of the keyword table with the @code{%keywordtable}
specifier.

The BNF file has two sections.  The first is the settings section, and
the second is the language definition, or list of semantic rules.

@menu
* Settings::                    Setup for a language
* Rules::                       Create rules to parse a language
* Optional Lambda Expression::  Actions to take when a rule is matched
* Examples::                    Simple Samples
* Style Guide::                 What the tokens mean, and how to use them.

@end menu

@node Settings, Rules, BNF conversion, BNF conversion
@section Settings

A setting is a keyword starting with a @asis{%}.  (This syntax is taken
from yacc and bison. @xref{(bison)}.)

There are several settings that can be made in the settings section.
They are:

@deffn Setting %start <nonterminal>
Specify an alternative to @code{bovine-toplevel}.  (See below)
@end deffn

@deffn Setting %scopestart <nonterminal>
Specify an alternative to @code{bovine-inner-scope}.
@end deffn

@deffn Setting %outputfile <filename>
Required.  Specifies the file into which this files output is stored.
@end deffn

@deffn Setting %parsetable <lisp-variable-name>
Required.  Specifies a lisp variable into which the output is stored.
@end deffn

@deffn Setting %setupfunction <lisp-function-name>
Required.  Name of a function into which setup code is to be inserted.
@end deffn

@deffn Setting %keywordtable <lisp-variable-name>
Required if there are @code{%token} keywords.
Specifies a lisp variable into which the output of a keyword table is
stored.  This obarray is used to turn symbols into keywords when applicable.
@end deffn

@deffn Setting %token <name> "<text>"
Optional.  Specify a new token @var{NAME}.  This is added to a lexical
keyword list using @var{TEXT}.  The symbol is then converted into a new
lexical terminal.  This requires that the @code{%keywordtable} specified
variable is available in the file specified by @code{%outputfile}.
@end deffn

@deffn Setting %token <name> type "<text>"
Optional.  Specify a new token @var{NAME}.  It is made from an existing
lexical token of type @var{TYPE}.  @var{TEXT} is a string which will be
matched explicitly.  @var{NAME} can be used in match rules as though they were
flex tokens, but are converted back to @var{TYPE} "text" internally.
@end deffn

@deffn Setting %put <NAME> symbol <VALUE>
@deffnx Setting %put <NAME> ( symbol1 <VALUE1> symbol2 <VALUE2> ... )
@deffnx Setting %put ( <NAME1> <NAME2>...) symbol <VALUE>
Tokens created without a type are considered keywords, and placed in a
keyword table.  Use @code{%put} to apply properties to that keyword.
(see @ref{Lexing}).
@end deffn

@deffn Setting %languagemode <lisp-function-name>
@deffnx Setting %languagemode ( <lisp-function-name1> <lisp-function-name2> ... )
Optional.  Specifies the Emacs major mode associated with the language
being specified.  When the language is converted, all buffers of this
mode will get the new table installed.
@end deffn

@deffn Setting %quotemode backquote
Optional.  Specifies how symbol quoting is handled in the Optional
Lambda Expressions. (See below)
@end deffn

@deffn Setting {%( <lisp-expression> )%}
Specify setup code to be inserted into the @code{%setupfunction}.
It will be inserted between two specifier strings, or added to
the end of the function.
@end deffn

When working inside @code{%( ... )%} tokens, any lisp expression can be
entered which will be placed inside the setup function.  In general, you
probably want to set variables that tell Semantic and related tools how
the language works.

Here are some variables that control how different programs will work
with your language.

@defvar semantic-flex-depth
Default flexing depth.
This specifies how many lists to create tokens in.
@end defvar

@defvar semantic-number-expression
Regular expression for matching a number.
If this value is @code{nil}, no number extraction is done during lex.
Symbols which match this expression are returned as @code{number}
tokens instead of @code{symbol} tokens.

The default value for this variable should work in most languages.
@end defvar

@defvar semantic-flex-extensions
Buffer local extensions to the lexical analyzer.
This should contain an alist with a key of a regex and a data element of
a function.  The function should both move point, and return a lexical
token of the form:
@example
( TYPE START . END)
@end example

@code{nil} is also a valid return.
@var{TYPE} can be any type of symbol, as long as it doesn't occur as a
nonterminal in the language definition.
@end defvar

@defvar semantic-flex-syntax-modifications
Updates to the syntax table for this buffer.
These changes are active only while this file is being flexed.
This is a list where each element is of the form:
@example
(CHAR CLASS)
@end example
Where @var{CHAR} is the char passed to @dfn{modify-syntax-entry},
and @var{CLASS} is the string also passed to @dfn{modify-syntax-entry} to define
what class of syntax @var{CHAR} is.
@end defvar

@defvar semantic-flex-enable-newlines
When flexing, report @code{'newlines} as syntactic elements.
Useful for languages where the newline is a special case terminator.
Only set this on a per mode basis, not globally.
@end defvar

@defvar semantic-ignore-comments
Default comment handling.
@code{t} means to strip comments when flexing.  @code{Nil} means to keep comments
as part of the token stream.
@end defvar

@defvar semantic-symbol->name-assoc-list
Association between symbols returned, and a string.
The string is used to represent a group of objects of the given type.
It is sometimes useful for a language to use a different string
in place of the default, even though that language will still
return a symbol.  For example, Java return's includes, but the
string can be replaced with @code{Imports}.
@end defvar

@defvar semantic-case-fold
Value for @code{case-fold-search} when parsing.
@end defvar

@defvar semantic-expand-nonterminal
Function to call for each nonterminal production.
Return a list of non-terminals derived from the first argument, or @code{nil}
if it does not need to be expanded.
Languages with compound definitions should use this function to expand
from one compound symbol into several.  For example, in @var{C} the
definition
@example
int a, b;
@end example
is easily parsed into one token, but represents multiple variables.  A
functions should be written which takes this compound token and turns
it into two tokens, one for @var{A}, and the other for @var{B}.

Within the language definition (the @file{.bnf} sources), it is often
useful to set the NAME slot of a token with a list of items that
distinguish each element in the compound definition.

This list can then be detected by the function set in
@code{semantic-expand-nonterminal} to create multiple tokens.
This function has one additional duty of managing the overlays created
by semantic.  It is possible to use the single overlay in the compound
token for all your tokens, but this can pose problems identifying
all tokens covering a given definition.

Please see @file{semantic-java.el} for an example of managing overlays
when expanding a token into multiple definitions.
@end defvar

@defvar semantic-override-table
Buffer local semantic function overrides alist.
These overrides provide a hook for a `major-mode' to override specific
behaviors with respect to generated semantic toplevel nonterminals and
things that these non-terminals are useful for.
Each element must be of the form: (SYM . FUN)
where SYM is the symbol to override, and FUN is the function to
override it with.

Available override symbols:

@multitable {@code{abbreviate-nonterminal}} {(token & nosnarf)} {Find token in buffer}
@item SYMBOL                 @tab PARAMETERS        @tab DESCRIPTION
@item find-dependency        @tab (token)           @tab Find the dependency file
@item find-nonterminal       @tab (token & parent)  @tab Find token in buffer.
@item find-documentation     @tab (token & nosnarf) @tab Find doc comments.
@item abbreviate-nonterminal @tab (token & parent)  @tab Return summary string.
@item summarize-nonterminal  @tab (token & parent)  @tab Return summary string.
@item prototype-nonterminal  @tab (token)           @tab Return a prototype string.
@item concise-prototype-nonterminal'     @tab (tok & parent color) @tab Return a concise prototype string.
@item uml-abbreviate-nonterminal'        @tab (tok & parent color) @tab Return a UML standard abbreviation string.
@item uml-prototype-nonterminal'         @tab (tok & parent color) @tab Return a UML like prototype string.
@item uml-concise-prototype-nonterminal' @tab (tok & parent color) @tab Return a UML like concise prototype string.

@item prototype-file         @tab (buffer)          @tab Return a file in which prototypes are placed
@item nonterminal-children   @tab (token)           @tab Return first rate children. These are children which may contain overlays.
@item nonterminal-external-member-parent @tab (token) @tab Parent of TOKEN
@item nonterminal-external-member-p @tab (parent token) @tab Non nil if TOKEN has PARENT, but is not in PARENT.
@item nonterminal-external-member-children @tab (token & usedb) @tab Get all external children of TOKEN.
@item nonterminal-protection @tab (token & parent)  @tab Return protection as a symbol.
@item nonterminal-abstract   @tab (token & parent)  @tab Return if TOKEN is abstract.
@item nonterminal-leaf       @tab (token & parent)  @tab Return if TOKEN is leaf.
@item nonterminal-static     @tab (token & parent)  @tab Return if TOKEN is static.
@item beginning-of-context   @tab (& point)         @tab Move to the beginning of the
					                 current context.
@item end-of-context         @tab (& point)         @tab Move to the end of the
					                 current context.
@item up-context             @tab (& point)         @tab Move up one context level.
@item get-local-variables    @tab (& point)         @tab Get local variables.
@item get-all-local-variables@tab (& point)         @tab Get all local variables.
@item get-local-arguments    @tab (& point)         @tab Get arguments to this function.

@item end-of-command         @tab                   @tab Move to the end of the current
                                                         command
@item beginning-of-command   @tab                   @tab Move to the beginning of the
                                                         current command
@item ctxt-current-symbol    @tab (& point)         @tab List of related symbols.
@item ctxt-current-assignment@tab (& point)         @tab Variable being assigned to.
@item ctxt-current-function  @tab (& point)         @tab Function being called at point.
@item ctxt-current-argument  @tab (& point)         @tab The index to the argument of
                                                         the current function the cursor
                                                         is in.
@end multitable

Parameters mean:

@table @code
@item &
Following parameters are optional
@item buffer
The buffer in which a token was found.
@item token
The nonterminal token we are doing stuff with
@item parent
If a TOKEN is stripped (of positional information) then this will be the
parent token which should have positional information in it.
@end table

@end defvar

@defvar semantic-type-relation-separator-character
Character strings used to separation a parent/child relationship.
This list of strings are used for displaying or finding separators
in variable field dereferencing.  The first character will be used for
display.  In @var{C}, a type field is separated like this: ``type.field''
thus, the character is a ``.''.  In @var{C}, and additional value of ``->''
would be in the list, so that ``type->field'' could be found.
@end defvar


@defvar semantic-dependency-include-path
Defines the include path used when searching for files.
This should be a list of directories to search which is specific to
the file being included.
This variable can also be set to a single function.  If it is a
function, it will be called with one arguments, the file to find as a
string, and  it should return the full path to that file, or nil.
@end defvar

This configures Imenu to use semantic parsing.

@defvar imenu-create-index-function
The function to use for creating a buffer index.

It should be a function that takes no arguments and returns an index
of the current buffer as an alist.

Simple elements in the alist look like @samp{(INDEX-NAME . INDEX-POSITION)}.
Special elements look like @samp{(INDEX-NAME INDEX-POSITION FUNCTION ARGUMENTS...)}.
A nested sub-alist element looks like (INDEX-NAME SUB-ALIST).
The function @code{imenu--subalist-p} tests an element and returns t
if it is a sub-alist.

This function is called within a @code{save-excursion}.

The variable is buffer-local.
@end defvar


These are specific to the document tool.

@table @code
@item document-comment-start
Comment start string.
@item document-comment-line-prefix
Comment prefix string.  Used at the beginning of each line.
@item document-comment-end
Comment end string.
@end table

@node Rules, Optional Lambda Expression, Settings, BNF conversion
@section Rules

Writing the rules should be very similar to bison for basic syntax.
Each rule is of the form

@example
RESULT : MATCH1 (optional-lambda-expression)
       | MATCH2 (optional-lambda-expression)
       ;
@end example

@var{RESULT} is a non-terminal, or a token synthesized in your grammar.
@var{MATCH} is a list of elements that are to be matched if @var{RESULT}
is to be made.  The optional lambda expression is a list containing
simplified rules for concocting the parse tree.

In bison, each time an element of a @var{MATCH} is found, it is
"shifted" onto the parser stack.  (The stack of matched elements.)  When
all of @var{MATCH1}'s elements have been matched, it is "reduced" to
@var{RESULT}.  @xref{(bison)Algorithm}.

The first @var{RESULT} written into your language specification should
be @code{bovine-toplevel}, or the symbol specified with @code{%start}.
When starting a parse for a file, this is the default token iterated
over.  You can use any token you want in place of @code{bovine-toplevel}
if you specify what that nonterminal will be with a @code{%start} token
in the settings section.

@var{MATCH} is made up of symbols and strings.  A symbol such as
@code{foo} means that a syntactic token of type @code{foo} must be
matched.  A string in the mix means that the previous symbol must have
the additional constraint of exactly matching it.  Thus, the
combination:

@example
symbol "moose"
@end example

means that a symbol must first be encountered, and then it must
@code{string-match "moose"}.  Be especially careful to remember that the
string is a regular expression.  The code:

@example
punctuation "."
@end example

will match any punctuation.

For the above example in bison, a LEX rule would be used to create a new
token @var{MOOSE}.  In this case, the @var{MOOSE} token would appear.
For the bovinator, this task was mixed into the language definition to
simplify implementation, though Bison's technique is more efficient.

To make a symbol match explicitly for keywords, for example, you can use
the @code{%token} command in the settings section to create new symbols.

@example
%token MOOSE "moose"

find_a_moose: MOOSE
            ;
@end example

will match ``moose'' explicitly, unlike the previous example where moose
need only appear in the symbol.  This is because ``moose'' will be
converted to @var{MOOSE} in the lexical analysis stage.  Thus the symbol
@var{MOOSE} won't be available any other way.

If we specify our token in this way:

@example
%token MOOSE symbol "moose"

find_a_moose: MOOSE
            ;
@end example

then @code{MOOSE} will match the string "moose" explicitly, but it won't
do so at the lexical level, allowing use of the text "moose" in other
forms of regular expressions.

Non symbol tokens are also allowed.  For example:

@example
%token PERIOD punctuation "."

filename : symbol PERIOD symbol
         ;
@end example

will explicitly match one period when used in the above rule.

@xref{Default syntactic tokens}.


@node Optional Lambda Expression, Examples, Rules, BNF conversion
@section Optional Lambda Expressions

The OLE (Optional Lambda Expression) is converted into a bovine lambda
(see @xref{Bovinating}.) This lambda has special short-cuts to simplify
reading the Emacs BNF definition.  An OLE like this:

@example
( $1 )
@end example

results in a lambda return which consists entirely of the string
or object found by matching the first (zeroth) element of match.
An OLE like this:

@example
( ,(foo $1) )
@end example

executes `foo' on the first argument, and then splices its return
into the return list whereas:

@example
( (foo $1) )
@end example

executes foo, and that is placed in the return list.

Here are other things that can appear inline:
@table @code
@item $1
the first object matched.
@item ,$1
the first object spliced into the list (assuming it is a list from a
non-terminal)
@item '$1
the first object matched, placed in a list.  i.e. ( $1 )
@item foo
the symbol foo (exactly as displayed)
@item (foo)
a function call to foo which is stuck into the return list.
@item ,(foo)
a function call to foo which is spliced into the return list.
@item '(foo)
a function call to foo which is stuck into the return list in a list.
@item (EXPAND $1 nonterminal depth)
a list starting with EXPAND performs a recursive parse on the token
passed to it (represented by $1 above.)  The semantic list is a common
token to expand, as there are often interesting things in the list.
The @var{nonterminal} is a symbol in your table which the bovinator will
start with when parsing.  @var{nonterminal}'s definition is the same as
any other nonterminal.  @var{depth} should be at least 1 when
descending into a semantic list.
@item (EXPANDFULL $1 nonterminal depth)
is like EXPAND, except that the parser will iterate over
@var{nonterminal} until there are no more matches.  (The same way the
parser iterates over @code{bovine-toplevel}. This lets you have
much simpler rules in this specific case, and also lets you have
positional information in the returned tokens, and error skipping.
@item (ASSOC symbol1 value1 symbol2 value2 ... )
This is used for creating an association list.  Each @var{SYMBOL} is
included in the list if the associated @var{VALUE} is non-nil.  While
the items are all listed explicitly, the created structure is an
association list of the form:
@example
( ( symbol1 . value1) (symbol2 . value2) ... )
@end example
@end table

If the symbol @code{%quotemode backquote} is specified, then use
@code{,@@} to splice a list in, and @code{,} to evaluate the expression.
This lets you send @code{$1} as a symbol into a list instead of having
it expanded inline.

@node Examples, Style Guide , Optional Lambda Expression, BNF conversion
@section Examples

The rule:

@example
SYMBOL : symbol
@end example

is equivalent to

@example
SYMBOL : symbol
         ( $1 )
@end example

which, if it matched the string "A", would return

@example
( "A" )
@end example

If this rule were used like this:

@example
ASSIGN: SYMBOL punctuation "=" SYMBOL
        ( $1 $3 )
@end example

it would match "A=B", and return

@example
( ("A") ("B") )
@end example

The letters A and B come back in lists because SYMBOL is a nonterminal,
not an actual lexical element.

to get a better result with nonterminals, use @asis{,} to splice lists
in like this;

@example
ASSIGN: SYMBOL punctuation "=" SYMBOL
        ( ,$1 ,$3 )
@end example

which would return

@example
( "A" "B" )
@end example

@node Style Guide ,  , Examples, BNF conversion
@section Semantic Token Style Guide

In order for a generalized program using Semantic to work with
multiple languages, it is important to have a consistent meaning for
the contents of the tokens returned.  The variable
@code{semantic-toplevel-bovine-table} is documented with the complete
list of a tokens that a functional or OO language may use.  While any
given language is free to create their own tokens, such a language
definition would not produce a stream of tokens usable by a
generalized tool.

@section Minimum Requirements

In general, all tokens returned from a parser should be generated with
the following form:

@example
("NAME" type-symbol ... "DOCSTRING" PROPERTIES OVERLAY)
@end example

@var{NAME} and @var{type-symbol} are the only syntactic elements of a
nonterminal which are guaranteed to exist.  This means that a parser
which uses @code{nil} for either of these two slots, or some value
which is not type consistent is wrong.

@var{NAME} is also guaranteed to be a string.  This string represents
the name of the nonterminal, usually a named definition which the
language will use elsewhere as a reference to the syntactic element
found.

@var{type-symbol} is a symbol representing the type of the
nonterminal.  Valid @var{type-symbol}s can be anything, as long is it
is an Emacs Lisp symbol.

@var{DOCSTRING} is a required slot in the nonterminal, but can be
nil.  Some languages have the documentation saved as a comment
nearby.  In these cases, DOCSTRING is nil, and the function
`semantic-find-documentation'.

@var{PROPERTIES} is a slot generated by the semantic parser harness,
and need not be provided by a language author.  Programmatically access
nonterminal properties with @code{semantic-token-put} and
@code{semantic-token-get} to access properties.

@var{OVERLAY} represents positional information for this token.  It is
automatically generated by the semantic parser harness, and need not
be provided by the language author, unless they provide a nonterminal
expansion function via @code{semantic-expand-nonterminal}.

The @var{OVERLAY} property is accessed via several functions returning
the beginning, end, and buffer of a token.  Use these functions unless
the overlay is really needed (see @ref{Token Queries}).  Depending on the
overlay in a program can be dangerous because sometimes the overlay is
replaced with an integer pair
@example
[ START END ]
@end example
when the buffer the token belongs to is not in memory.  This happens
when a using has activated the Semantic Database @ref{semanticdb}.

@section Nonterminals for Functional Languages.

If a parser produces tokens for a functional language, then the
following token formats are available.

@table @asis
@item Variable
@itemx @code{("NAME" variable "TYPE" DEFAULT-VALUE EXTRA-SPEC}
@itemx @code{        "DOCSTRING" PROPERTIES OVERLAY)}
@var{TYPE} is a string representing the type of this variable.
@var{TYPE} can be @code{nil} for untyped languages.  Languages which
support variable declarations without a type (Such as C) should supply
a string representing the default type for that language.

@var{DEFAULT-VALUE} can be a string, or something pre-parsed and
language specific.  Hopefully this slot will be better defined in
future versions of Semantic.

@var{EXTRA-SPEC} are extra specifiers.  See below.


@item Function
@itemx @code{ ("NAME" function "TYPE" ( ARG-LIST ) EXTRA-SPEC}
@itemx @code{          "DOCSTRING" PROPERTIES OVERLAY)}
@var{TYPE} is a string representing the return type of this function
or method.  @var{type} can be @code{nil} for untyped languages, or for
procedures in languages which support functions with no return data.
See above for more.

@var{ARG-LIST} is a list of arguments passed to this function.
Each element in the arg list can be one of the following:
@table @asis
@item Semantic Token
A full semantic token with positional information.
@item A partial semantic token
Partial tokens may contain the @var{NAME} slot, @var{token-symbol},
and possibly a @var{TYPE}.
@item String
A string representing the name of the argument.  Common in untyped
languages.
@end table


@item Type Declaration
@itemx @code{ ("NAME" type "TYPE" ( PART-LIST ) ( PARENTS ) EXTRA-SPEC}
@itemx @code{          "DOCSTRING" PROPERTIES OVERLAY)}
@var{TYPE} a string representing the of the type, such as (in C)
``struct'', ``union'', ``enum'', ``typedef'', or ``class''.
The @var{TYPE} for a type token should not be nil, as even untyped
languages with structures have type types.

@var{PART-LIST} is the list of individual entries inside compound
types.  Structures, for example, can contain several fields which can
be represented as variables.  Valid entries in a @var{PART-LIST} are:
@table @asis
@item Semantic Token
A full semantic token with positional information.
@item A partial semantic token
Partial tokens may contain the @var{NAME} slot, @var{token-symbol},
and possibly a @var{TYPE}.
@item String
A string representing the name of the slot or field.  Common in untyped
languages.
@end table

@var{PARENTS} represents a list of parents of this type.  Parents are used
in two situations.
@table @asis
@item Inheritance
For types which inherit from other types of the same type-type (Such
as classes).
@item Aliases
For types which are aliases of other types, the parent type is the
type being aliased.  The Types' type is the command specifying that it
is an alias (Such as ``typedef'' in C or C++).
@end table

The structure of the @var{PARENTS} list is of this form:
@example
( EXPLICIT-PARENTS . INTERFACE-PARENTS)
@end example
@var{EXPLICIT-PARENTS} can be a single string (Just one parent) or a
list of parents (in a multiple inheritance situation.  It can also be
nil.

@var{INTERFACE-PARENTS} is a list of strings representing the names of
all INTERFACES, or abstract classes inherited from.  It can also be
nil.

This slot can be interesting because the form:
@example
( nil "string")
@end example
is a valid parent where there is no explicit parent, and only an
interface.


@item Include files
@itemx @code{("FILE" include SYSTEM "DOCSTRING" PROPERTIES OVERLAY)}
A statement which gets additional definitions from outside the current
file, such as an @code{#include} statement in C.
In this case, instead of @var{NAME}, a @var{FILE} is specified.
@var{FILE} can be a subset of the actual file to be loaded.

@var{SYSTEM} is true if this include is part of a set of system
includes.  This field isn't currently being used and may be
eliminated.


@item Package & Provide statements
@itemx @code{("NAME" package DETAIL "DOCSTRING" PROPERTIES OVERLAY)}
A statement which declares a given file is part of a package, such as
the Java @code{package} statement, or a @code{provide} in Emacs Lisp.

@var{DETAIL} might be an associated file name, or some other language
specific bit of information.
@end table

@section Extra Specifiers

Some default token types have a slot EXTRA-SPEC, for extra specifiers.
These specifiers provide additional details not commonly used, or not
available in all languages.  This list is an alist, and if a given key
is nil, it is not in the list, saving space.  Some valid extra
specifiers are:

@table @code
@item (parent .  "text")
Name of a parent type/class.  This is not the same as a parent for a
type.  In C++ and CLOS allow the creation of a function outside the
body of that class.  Such functions will set the @var{parent}
specifier to a plain text string which is the name of that parent.

@item (dereference .  INT)
Number of levels of dereference.
In C, the number of array dimensions.

@item (pointer . INT)
Number of levels of pointers.
In C, the number of @code{*} characters.

@item (typemodifiers .  ( "text" ... ))
Keyword modifiers for a type.  In C, such words would include
@code{register'} and @code{volatile'}

@item (suffix . "text")
Suffix information for a variable.  Not currently used.

@item (const .  t)
This exists if the variable or function return value is constant.

@item (throws .  ( "text" ... ))
For functions or methods in languages that support typed signal
throwing, this is a list of exceptions that can be thrown.

@item (destructor . t)
This exists for functions which are destructor methods in a class
definition.  In C++, a destructor's name excludes the ~ character.  When
producing the name of the function, the ~ is added back in.

@item (constructor . t)
This exists for functions which are constructors in a class
definition.  In C++ this is t when the name of this function is
the same as the name of the parent class.

@item (user-visible . t)
For functions in interpreted languages such as Emacs Lisp, this
signals that a function or variable is user visible.  In Emacs Lisp,
this means a function is @dfn{interactive}.

@item (prototype . t)
For functions or variables that are not declared locally, a prototype
is something that will define that function or variable for use.
In C, the term represents prototypes generally used in header files.
In Emacs Lisp, the @code{autoload} statement creates prototypes.
@end table

@node Compiling, Debugging, BNF conversion, Top
@chapter Compiling a language file with the bovinator

From a program you can use the function @code{semantic-bovinate-toplevel}.
This function takes one optional parameter specifying if the cache
should be refreshed.  By default, the cached results of the last parse
are always used.  Specifying that the cache should be checked will cause
it to be flushed if it is out of date.

Another function you can use is @code{semantic-bovinate-nonterminal}.
This command takes a token stream returned by the function
@code{semantic-flex} followed by a DEPTH (as above).  This takes an
additional optional argument of NONTERMINAL which is the nonterminal in
your table it is to start parsing with.

@deffn Command bovinate &optional clear
Bovinate the current buffer.  Show output in a temp buffer.
Optional argument @var{CLEAR} will clear the cache before bovinating.
@end deffn

@deffn Command semantic-clear-toplevel-cache
Clear the toplevel bovine cache for the current buffer.
Clearing the cache will force a complete reparse next time a token
stream is requested.
@end deffn

@defun semantic-bovinate-toplevel &optional checkcache
Bovinate the entire current buffer.
If the optional argument @var{CHECKCACHE} is non-@code{nil}, then flush the cache iff
there has been a size change.
@end defun

@node Debugging, Programming, Compiling, Top
@comment  node-name,  next,  previous,  up
@chapter Debugging

Writing language files using BNF is significantly easier than writing
then using regular expressions in a functional manner.  Debugging
them, however, can still prove challenging.

There are two ways to debug a language definition if it is not
behaving as expected.  One way is to debug against the source @file{.bnf}
file.  The second is to debug against the lisp table created from the
@file{.bnf} source, or perhaps written by hand.

If your language definition was written in BNF notation, debugging is
quite easy.  The command @code{bovinate-debug} will start you off.

@deffn Command bovinate-debug
Bovinate the current buffer and run in debug mode.
@end deffn

If you prefer debugging against the Lisp table, find the table in a
buffer, place the cursor in it, and use the command
@code{semantic-bovinate-debug-set-table} in it.

@deffn Command semantic-bovinate-debug-set-table &optional clear
Set the table for the next debug to be here.
Optional argument @var{CLEAR} to unset the debug table.
@end deffn

After the table is set, the @code{bovinate-debug} command can be run
at any time for the given language.

While debugging, two windows are visible.  One window shows the file
being parsed, and the syntactic token being tested is highlighted.  The
second window shows the table being used (either in the BNF source, or
the Lisp table) with the current rule highlighted.  The cursor will
sit on the specific match rule being tested against.

In the minibuffer, a brief summary of the current situation is
listed.  The first element is the syntactic token which is a list of
the form:

@example
(TYPE START . END)
@end example

The rest of the display is a list of all strings collected for the
currently tested rule.  Each time a new rule is entered, the list is
restarted.  Upon returning from a rule into a previous match list, the
previous match list is restored, with the production of the dependent
rule in the list.

Use @kbd{C-g} to stop debugging.  There are no commands for any
fancier types of debugging.

@node Programming, Current Context, Debugging, Top
@comment  node-name,  next,  previous,  up
@chapter Programming

Once a source file has been parsed, the following APIs can be used to
write programs that use the token stream most effectively.

@menu
* Token Queries::               Getting info about a parsed token (nonterminal).
* Nonterminal Streams::         Working with streams of nonterminals.
* Nonterminals at point::       Finding nonterminals at point.
* Nonterminal Sorting::         Reorganizing streams.
* Nonterminal Completion::      Completing read functions.
* Override Methods::            Language dependent functions covering
                                conversion to text strings, language dependent
                                queries and local context information
* Parser Hooks::                How to know when tags change.
* Example Programs::            Simple programming examples.
@end menu

@node Token Queries, Nonterminal Streams, Programming, Programming
@comment  node-name,  next,  previous,  up
@section Token Queries

When writing programs that use the bovinator, the following functions
are needed to find get details out of a nonterminal.

@defun semantic-equivalent-tokens-p token1 token2
Compare @var{TOKEN1} and @var{TOKEN2} and return non-@code{nil} if they are equivalent.
Use @dfn{eq} to test of two tokens are the same.  Use this function if tokens
are being copied and regrouped to test for if two tokens represent the same
thing, but may be constructed of different cons cells.
@end defun

@defun semantic-token-token token
Retrieve from @var{TOKEN} the token identifier.
i.e., the symbol @code{'variable}, @code{'function}, @code{'type}, or other.
@end defun

@defun semantic-token-name token
Retrieve the name of @var{TOKEN}.
@end defun

@defun semantic-token-docstring token &optional buffer
Retrieve the documentation of @var{TOKEN}.
Optional argument @var{BUFFER} indicates where to get the text from.
If not provided, then only the @var{POSITION} can be provided.
@end defun

@defun semantic-token-overlay token
Retrieve the @var{OVERLAY} part of @var{TOKEN}.
The returned item may be an overlay or an unloaded buffer representation.
@end defun

@defun semantic-token-extent token
Retrieve the extent (@var{START} @var{END}) of @var{TOKEN}.
@end defun

@defun semantic-token-start token
Retrieve the start location of @var{TOKEN}.
@end defun

@defun semantic-token-end token
Retrieve the end location of @var{TOKEN}.
@end defun

@defun semantic-token-type token
Retrieve the type of @var{TOKEN}.
@end defun

@defun semantic-token-put token property value
On @var{token}, set @var{property} to @var{value}.
@end defun

@defun semantic-token-get token property
For @var{token} get the value of @var{property}.
@end defun

@defun semantic-token-extra-spec token spec
Retrieve a specifier for the variable @var{TOKEN}.
@var{SPC} is the symbol whose modifier value to get.
This function can get specifiers from any type of @var{TOKEN}.
Do not use this function if you know what type of token you are dereferencing.
Instead, use the function specific to that token type.  It will be faster.
@end defun

@defun semantic-token-type-parts token
Retrieve the parts of the type @var{TOKEN}.
@end defun

@defun semantic-token-type-parent token
Retrieve the parent of the type @var{TOKEN}.
The return value is a list.  @var{A} value of @code{nil} means no parents.
The @dfn{car} of the list is either the parent class, or a list
of parent classes.  The @dfn{cdr} of the list is the list of
interfaces, or abstract classes which are parents of @var{TOKEN}.
@end defun

@defun semantic-token-type-parent-superclass token
Retrieve the parent super classes of type type @var{TOKEN}.
@end defun

@defun semantic-token-type-parent-implement token
Retrieve the parent interfaces of type type @var{TOKEN}.
@end defun

@defun semantic-token-type-modifiers token
Retrieve the type modifiers for the type @var{TOKEN}.
@end defun

@defun semantic-token-type-extra-specs token
Retrieve the extra specifiers for the type @var{TOKEN}.
@end defun

@defun semantic-token-type-extra-spec token spec
Retrieve a extra specifier for the type @var{TOKEN}.
@var{SPEC} is the symbol whose modifier value to get.
@end defun

@defun semantic-token-function-args token
Retrieve the arguments of the function @var{TOKEN}.
@end defun

@defun semantic-token-function-modifiers token
Retrieve the type modifiers of the function @var{TOKEN}.
@end defun

@defun semantic-token-function-destructor token
Non-@code{nil} if @var{TOKEN} is a destructor function.
@end defun

@defun semantic-token-function-extra-specs token
Retrieve the extra specifiers of the function @var{TOKEN}.
@end defun

@defun semantic-token-function-extra-spec token spec
Retrieve a specifier for the function @var{TOKEN}.
@var{SPEC} is a symbol whose specifier value to get.
@end defun

@defun semantic-token-function-throws token
Retrieve the throws signal of the function @var{TOKEN}.
This is an optional field, and returns @code{nil} if it doesn't exist.
@end defun

@defun semantic-token-function-parent token
The parent of the function @var{TOKEN}.
A function has a parent if it is a method of a class, and if the
function does not appear in body of its parent class.
@end defun

@defun semantic-token-variable-const token
Retrieve the status of constantness from the variable @var{TOKEN}.
@end defun

@defun semantic-token-variable-default token
Retrieve the default value of the variable @var{TOKEN}.
@end defun

@defun semantic-token-variable-modifiers token
Retrieve type modifiers for the variable @var{TOKEN}.
@end defun

@defun semantic-token-variable-extra-specs token
Retrieve extra specifiers for the variable @var{TOKEN}.
@end defun

@defun semantic-token-variable-extra-spec token spec
Retrieve a specifier value for the variable @var{TOKEN}.
@var{SPEC} is the symbol whose specifier value to get.
@end defun

@defun semantic-token-include-system token
Retrieve the flag indicating if the include @var{TOKEN} is a system include.
@end defun

For override methods that query a token, see @xref{Token Details}.

@node Nonterminal Streams, Nonterminals at point, Token Queries, Programming
@section Nonterminal streams

These functions take some key, and returns information found inside
the nonterminal stream.  Some will return one token (the first
matching item found.)  Others will return a list of all items matching
a given criterion.  All these functions work regardless of a buffer
being in memory or not.

@defun semantic-find-nonterminal-by-name name streamorbuffer &optional search-parts search-include
Find a nonterminal @var{NAME} within @var{STREAMORBUFFER}.  @var{NAME} is a string.
If @var{SEARCH-PARTS} is non-@code{nil}, search children of tokens.
If @var{SEARCH-INCLUDE} is non-@code{nil}, search include files.
@end defun

@defun semantic-find-nonterminal-by-property property value streamorbuffer &optional search-parts search-includes
Find all nonterminals with @var{PROPERTY} equal to @var{VALUE} in @var{STREAMORBUFFER}.
Properties can be added with @dfn{semantic-token-put}.
Optional argument @var{SEARCH-PARTS} and @var{SEARCH-INCLUDES} are passed to
@dfn{semantic-find-nonterminal-by-function}.
@end defun

@defun semantic-find-nonterminal-by-extra-spec spec streamorbuffer &optional search-parts search-includes
Find all nonterminals with a given @var{SPEC} in @var{STREAMORBUFFER}.
@var{SPEC} is a symbol key into the modifiers association list.
Optional argument @var{SEARCH-PARTS} and @var{SEARCH-INCLUDES} are passed to
@dfn{semantic-find-nonterminal-by-function}.
@end defun

@defun semantic-find-nonterminal-by-extra-spec-value spec value streamorbuffer &optional search-parts search-includes
Find all nonterminals with a given @var{SPEC} equal to @var{VALUE} in @var{STREAMORBUFFER}.
@var{SPEC} is a symbol key into the modifiers association list.
@var{VALUE} is the value that @var{SPEC} should match.
Optional argument @var{SEARCH-PARTS} and @var{SEARCH-INCLUDES} are passed to
@dfn{semantic-find-nonterminal-by-function}.
@end defun

@defun semantic-find-nonterminal-by-position position streamorbuffer &optional nomedian
Find a nonterminal covering @var{POSITION} within @var{STREAMORBUFFER}.
@var{POSITION} is a number, or marker.  If @var{NOMEDIAN} is non-@code{nil}, don't do
the median calculation, and return nil.
@end defun

@defun semantic-find-innermost-nonterminal-by-position position streamorbuffer &optional nomedian
Find a list of nonterminals covering @var{POSITION} within @var{STREAMORBUFFER}.
@var{POSITION} is a number, or marker.  If @var{NOMEDIAN} is non-@code{nil}, don't do
the median calculation, and return nil.
This function will find the topmost item, and recurse until no more
details are available of findable.
@end defun

@defun semantic-find-nonterminal-by-token token streamorbuffer &optional search-parts search-includes
Find all nonterminals with a token @var{TOKEN} within @var{STREAMORBUFFER}.
@var{TOKEN} is a symbol representing the type of the tokens to find.
Optional argument @var{SEARCH-PARTS} and @var{SEARCH-INCLUDE} are passed to
@dfn{semantic-find-nonterminal-by-function}.
@end defun

@defun semantic-find-nonterminal-standard streamorbuffer &optional search-parts search-includes
Find all nonterminals in @var{STREAMORBUFFER} which define simple token types.
Optional argument @var{SEARCH-PARTS} and @var{SEARCH-INCLUDE} are passed to
@dfn{semantic-find-nonterminal-by-function}.
@end defun

@defun semantic-find-nonterminal-by-type type streamorbuffer &optional search-parts search-includes
Find all nonterminals with type @var{TYPE} within @var{STREAMORBUFFER}.
@var{TYPE} is a string which is the name of the type of the token returned.
Optional argument @var{SEARCH-PARTS} and @var{SEARCH-INCLUDES} are passed to
@dfn{semantic-find-nonterminal-by-function}.
@end defun

@defun semantic-find-nonterminal-by-function function streamorbuffer &optional search-parts search-includes
Find all nonterminals in which @var{FUNCTION} match within @var{STREAMORBUFFER}.
@var{FUNCTION} must return non-@code{nil} if an element of @var{STREAM} will be included
in the new list.

If optional argument @var{SEARCH-PARTS} is non-@code{nil}, all sub-parts of tokens
are searched.  The over-loadable function @dfn{semantic-nonterminal-children} is
used for the searching child lists.  If @var{SEARCH-PARTS} is the symbol
@code{'positiononly}, then only children that have positional information are
searched.

If @var{SEARCH-INCLUDES} is non-@code{nil}, then all include files are also
searched for matches.
@end defun

@defun semantic-find-nonterminal-by-function-first-match function streamorbuffer &optional search-parts search-includes
Find the first nonterminal which @var{FUNCTION} match within @var{STREAMORBUFFER}.
@var{FUNCTION} must return non-@code{nil} if an element of @var{STREAM} will be included
in the new list.
If optional argument @var{SEARCH-PARTS}, all sub-parts of tokens are searched.
The over-loadable function @dfn{semantic-nonterminal-children} is used for
searching.
If @var{SEARCH-INCLUDES} is non-@code{nil}, then all include files are also
searched for matches.
@end defun

@defun semantic-recursive-find-nonterminal-by-name name buffer
Recursively find the first occurrence of @var{NAME}.
Start search with @var{BUFFER}.  Recurse through all dependencies till found.
The return item is of the form (@var{BUFFER} @var{TOKEN}) where @var{BUFFER} is the buffer
in which @var{TOKEN} (the token found to match @var{NAME}) was found.
@end defun

@node Nonterminals at point, Nonterminal Sorting, Nonterminal Streams, Programming
@comment  node-name,  next,  previous,  up
@section Nonterminals at point

When you just want to get at a nonterminal the cursor is on, there is
a more efficient mechanism than using
@code{semantic-find-nonterminal-by-position}.  This mechanism
directly queries the overlays the parsing step leaves in the buffer.
This provides for very rapid retrieval of what function or variable
the cursor is currently in.
@refill

These functions query the current buffer's overlay system for tokens.

@defun semantic-find-nonterminal-by-overlay &optional positionormarker buffer
Find all nonterminals covering @var{POSITIONORMARKER} by using overlays.
If @var{POSITIONORMARKER} is @code{nil}, use the current point.
Optional @var{BUFFER} is used if @var{POSITIONORMARKER} is a number, otherwise the current
buffer is used.  This finds all tokens covering the specified position
by checking for all overlays covering the current spot.  They are then sorted
from largest to smallest via the start location.
@end defun

@defun semantic-find-nonterminal-by-overlay-in-region start end &optional buffer
Find all nonterminals which exist in whole or in part between @var{START} and @var{END}.
Uses overlays to determine position.
Optional @var{BUFFER} argument specifies the buffer to use.
@end defun

@defun semantic-current-nonterminal
Return the current nonterminal in the current buffer.
If there are more than one in the same location, return the
smallest token.
@end defun

@defun semantic-current-nonterminal-parent
Return the current nonterminals parent in the current buffer.
@var{A} token's parent would be a containing structure, such as a type
containing a field.  Return @code{nil} if there is no parent.
@end defun

@node Nonterminal Sorting, Nonterminal Completion, Nonterminals at point, Programming
@section Nonterminal sorting

Sometimes it is important to reorganize a token stream into a form
that is better for display to a user.  It is important to not use
functions with side effects when doing this, and that could effect
the token cache.

There are some existing utility functions which will reorganize the
token list for you.

@defun semantic-bucketize tokens &optional parent filter
Sort @var{TOKENS} into a group of buckets based on token type.
Unknown types are placed in a Misc bucket.
Type bucket names are defined by either `semantic-symbol->name-assoc-list'.
If @var{PARENT} is specified, then @var{TOKENS} belong to this @var{PARENT} in some way.
This will use `semantic-symbol->name-assoc-list-for-type-parts' to
generate bucket names.
Optional argument @var{FILTER} is a filter function to be applied to each bucket.
The filter function will take one argument, which is a list of tokens, and
may re-organize the list with side-effects.
@end defun

@defvar semantic-bucketize-token-token
Function used to get a symbol describing the class of a token.
This function must take one argument of a semantic token.
It should return a symbol found in `semantic-symbol->name-assoc-list'
which @dfn{semantic-bucketize} uses to bin up tokens.
To create new bins for an application augment
`semantic-symbol->name-assoc-list', and
`semantic-symbol->name-assoc-list-for-type-parts' in addition
to setting this variable (locally in your function).
@end defvar

@defun semantic-adopt-external-members tokens
Rebuild @var{TOKENS} so that externally defined members are regrouped.
Some languages such as @var{C}++ and @var{CLOS} permit the declaration of member
functions outside the definition of the class.  It is easier to study
the structure of a program when such methods are grouped together
more logically.

This function uses @dfn{semantic-nonterminal-external-member-p} to
determine when a potential child is an externally defined member.

Note: Applications which use this function must account for token
types which do not have a position, but have children which *do*
have positions.

Applications should use @code{semantic-mark-external-member-function}
to modify all tokens which are found as externally defined to some
type.  For example, changing the token type for generating extra
buckets with the bucket function.
@end defun

@defvar semantic-orphaned-member-metaparent-type
In @dfn{semantic-adopt-external-members}, the type of @code{'type} for metaparents.
@var{A} metaparent is a made-up type semantic token used to hold the child list
of orphaned members of a named type.
@end defvar

@defvar semantic-mark-external-member-function
Function called when an externally defined orphan is found.
Be default, the token is always marked with the @code{adopted} property.
This function should be locally bound by a program that needs
to add additional behaviors into the token list.
This function is called with one argument which is a shallow copy
of the token to be modified.  This function should return the
token (or a copy of it) which is then integrated into the
revised token list.
@end defvar

@node Nonterminal Completion, Override Methods, Nonterminal Sorting, Programming
@section Nonterminal completion

These functions provide ways reading the names of items in a buffer with
completion.


@defun semantic-read-symbol prompt &optional default stream filter
Read a symbol name from the user for the current buffer.
@var{PROMPT} is the prompt to use.
Optional arguments:
@var{DEFAULT} is the default choice.  If no default is given, one is read
from under point.
@var{STREAM} is the list of tokens to complete from.
@var{FILTER} is provides a filter on the types of things to complete.
@var{FILTER} must be a function to call on each element.  (See !!!
@end defun

@defun semantic-read-variable prompt &optional default stream
Read a variable name from the user for the current buffer.
@var{PROMPT} is the prompt to use.
Optional arguments:
@var{DEFAULT} is the default choice.  If no default is given, one is read
from under point.
@var{STREAM} is the list of tokens to complete from.
@end defun

@defun semantic-read-function prompt &optional default stream
Read a function name from the user for the current buffer.
@var{PROMPT} is the prompt to use.
Optional arguments:
@var{DEFAULT} is the default choice.  If no default is given, one is read
from under point.
@var{STREAM} is the list of tokens to complete from.
@end defun

@defun semantic-read-type prompt &optional default stream
Read a type name from the user for the current buffer.
@var{PROMPT} is the prompt to use.
Optional arguments:
@var{DEFAULT} is the default choice.  If no default is given, one is read
from under point.
@var{STREAM} is the list of tokens to complete from.
@end defun


@node Override Methods, Parser Hooks, Nonterminal Completion, Programming
@section Override Methods

These functions are called `override methods' because they provide
generic behaviors, which a given language can override.  For example,
finding a dependency file in Emacs lisp can be done with the
`locate-library' command (which overrides the default behavior.)  In C,
a dependency can be found by searching a generic search path which can
be passed in via a variable.

@menu
* Token->Text:: Converting Tokens into text strings
* Token Details:: Arbitrary token detail fetching
* Local Context:: Deriving information about a language specific local context.
* Making New Methods:: How to add your own methods for a tool
@end menu

@node Token->Text, Token Details, Override Methods, Override Methods
@subsection Token->Text

Any given token consists of Meta information which is best viewed in
some textual form.  This could be as simple as the token's name, or as
a prototype to be added to header file in C.  Not only are there
several default converters from a Token into text, but there is also
some convenient variables that can be used with them.  Use these
variables to allow options on output forms when displaying tokens in
your programs.

@defvar semantic-token->text-functions
List of functions which convert a token to text.
Each function must take the parameters @var{TOKEN} &optional @var{PARENT} @var{COLOR}.
@var{TOKEN} is the token to convert.
@var{PARENT} is a parent token or name which refers to the structure
or class which contains @var{TOKEN}.  @var{PARENT} is @var{NOT} a class which a @var{TOKEN}
would claim as a parent.
@var{COLOR} indicates that the generated text should be colored using
@code{font-lock}.
@end defvar

@defvar semantic-token->text-custom-list
@var{A} List used by customizable variables to choose a token to text function.
Use this variable in the @code{:type} field of a customizable variable.
@end defvar

Every token to text conversion function must take the same parameters,
which are @var{TOKEN}, the token to be converted, @var{PARENT}, the
containing parent (like a structure which contains a variable), and
@var{COLOR}, which is a flag specifying that color should be applied
to the returned string.

When creating, or using these strings, particularly with color, use
@dfn{concat} to build up larger strings instead of @dfn{format}.  This
will preserve text properties.

@defun semantic-name-nonterminal token &optional parent color
Return the name string describing @var{TOKEN}.
The name is the shortest possible representation.
Optional argument @var{PARENT} is the parent type if @var{TOKEN} is a detail.
Optional argument @var{COLOR} means highlight the prototype with font-lock colors.
@end defun

@defun semantic-summarize-nonterminal token &optional parent color
Summarize @var{TOKEN} in a reasonable way.
Optional argument @var{PARENT} is the parent type if @var{TOKEN} is a detail.
Optional argument @var{COLOR} means highlight the prototype with font-lock colors.
@end defun

@defun semantic-prototype-nonterminal token &optional parent color
Return a prototype for @var{TOKEN}.
This function should be overloaded, though it need not be used.
This is because it can be used to create code by language independent
tools.
Optional argument @var{PARENT} is the parent type if @var{TOKEN} is a detail.
Optional argument @var{COLOR} means highlight the prototype with font-lock colors.
@end defun

@defun semantic-prototype-file buffer
Return a file in which prototypes belonging to @var{BUFFER} should be placed.
Default behavior (if not overridden) looks for a token specifying the
prototype file, or the existence of an @var{EDE} variable indicating which
file prototypes belong in.
@end defun

@defun semantic-abbreviate-nonterminal token &optional parent color
Return an abbreviated string describing @var{TOKEN}.
The abbreviation is to be short, with possible symbols indicating
the type of token, or other information.
Optional argument @var{PARENT} is the parent type if @var{TOKEN} is a detail.
Optional argument @var{COLOR} means highlight the prototype with font-lock colors.
@end defun

@defun semantic-concise-prototype-nonterminal token &optional parent color
Return a concise prototype for @var{TOKEN}.
Optional argument @var{PARENT} is the parent type if @var{TOKEN} is a detail.
Optional argument @var{COLOR} means highlight the prototype with font-lock colors.
@end defun

@defun semantic-uml-abbreviate-nonterminal token &optional parent color
Return a @var{UML} style abbreviation for @var{TOKEN}.
Optional argument @var{PARENT} is the parent type if @var{TOKEN} is a detail.
Optional argument @var{COLOR} means highlight the prototype with font-lock colors.
@end defun

@node Token Details, Local Context, Token->Text, Override Methods
@subsection Token Details

These functions help derive information about tokens that may not
be obvious for non-traditional languages with their own token types.

@defun semantic-nonterminal-children token &optional positionalonly
Return the list of top level children belonging to @var{TOKEN}.
Children are any sub-tokens which may contain overlays.
The default behavior (if not overridden with @code{nonterminal-children}
is to return type parts for a type, and arguments for a function.

If optional argument @var{POSITIONALONLY} is non-@code{nil}, then only return valid
children if they contain positions.  Some languages may choose to create
lists of children without position/overlay information.

If this function is overridden, use @dfn{semantic-nonterminal-children-default}
to also include the default behavior, and merely extend your own.

Note for language authors:
  If a mode defines a language that has tokens in it with overlays that
should not be considered children, you should still return them with
this function.  If you do not, then token re-parsing, and database
saving will fail.
@end defun

@defun semantic-nonterminal-external-member-parent token
Return a parent for @var{TOKEN} when @var{TOKEN} is an external member.
@var{TOKEN} is an external member if it is defined at a toplevel and
has some sort of label defining a parent.  The parent return will
be a string.

The default behavior, if not overridden with
@code{nonterminal-external-member-parent} is get the @code{'parent} extra
specifier of @var{TOKEN}.

If this function is overridden, use
@dfn{semantic-nonterminal-external-member-parent-default} to also
include the default behavior, and merely extend your own.
@end defun

@defun semantic-nonterminal-external-member-p parent token
Return non-@code{nil} if @var{PARENT} is the parent of @var{TOKEN}.
@var{TOKEN} is an external member of @var{PARENT} when it is somehow tagged
as having @var{PARENT} as it's parent.

The default behavior, if not overridden with
@code{nonterminal-external-member-p} is to match @code{'parent} extra specifier in
the name of @var{TOKEN}.

If this function is overridden, use
@code{semantic-nonterminal-external-member-children-p-default} to also
include the default behavior, and merely extend your own.
@end defun

@defun semantic-nonterminal-external-member-children token &optional usedb
Return the list of children which are not *in* @var{TOKEN}.
If optional argument @var{USEDB} is non-@code{nil}, then also search files in
the Semantic Database.  If @var{USEDB} is a list of databases, search those
databases.

Children in this case are functions or types which are members of
@var{TOKEN}, such as the parts of a type, but which are not defined inside
the class.  @var{C}++ and @var{CLOS} both permit methods of a class to be defined
outside the bounds of the class' definition.

The default behavior, if not overridden with
@code{nonterminal-external-member-children} is to search using
@dfn{semantic-nonterminal-external-member-p} in all top level definitions
with a parent of @var{TOKEN}.

If this function is overridden, use
@dfn{semantic-nonterminal-external-member-children-default} to also
include the default behavior, and merely extend your own.
@end defun

@defun semantic-nonterminal-protection token &optional parent
Return protection information about @var{TOKEN} with optional @var{PARENT}.
This function returns on of the following symbols:
   @code{nil}      - No special protection.  Language dependent.
   @code{'public}  - Anyone can access this @var{TOKEN}.
   @code{'private} - Only methods in the local scope can access @var{TOKEN}.
   @code{'friend}  - Like private, except some outer scopes are allowed
              access to token.
Some languages may choose to provide additional return symbols specific
to themselves.  Use of this function should allow for this.

The default behavior (if not overridden with @code{nonterminal-protection}
is to return a symbol based on type modifiers.
@end defun

@defun semantic-nonterminal-abstract token &optional parent
Return non @code{nil} if @var{TOKEN} is abstract.
Optional @var{PARENT} is the parent token of @var{TOKEN}.
In @var{UML}, abstract methods and classes have special meaning and behavior
in how methods are overridden.  In @var{UML}, abstract methods are italicized.

The default behavior (if not overridden with @code{nonterminal-abstract}
is to return true if @code{abstract} is in the type modifiers.
@end defun

@defun semantic-nonterminal-leaf token &optional parent
Return non @code{nil} if @var{TOKEN} is leaf.
Optional @var{PARENT} is the parent token of @var{TOKEN}.
In @var{UML}, leaf methods and classes have special meaning and behavior.

The default behavior (if not overridden with @code{nonterminal-leaf}
is to return true if @code{leaf} is in the type modifiers.
@end defun

@defun semantic-nonterminal-static token &optional parent
Return non @code{nil} if @var{TOKEN} is static.
Optional @var{PARENT} is the parent token of @var{TOKEN}.
In @var{UML}, static methods and attributes mean that they are allocated
in the parent class, and are not instance specific.
@var{UML} notation specifies that @var{STATIC} entries are underlined.

The default behavior (if not overridden with @code{nonterminal-static}
is to return true if @code{static} is in the type modifiers.
@end defun

@defun semantic-find-dependency token
Find the filename represented from @var{TOKEN}.
@var{TOKEN} may be a stripped element, in which case @var{PARENT} specifies a
parent token that has positional information.
Depends on @code{semantic-dependency-include-path} for searching.  Always searches
`.' first, then searches additional paths.


@defvar semantic-dependency-include-path
Defines the include path used when searching for files.
This should be a list of directories to search which is specific to
the file being included.
This variable can also be set to a single function.  If it is a
function, it will be called with one arguments, the file to find as a
string, and  it should return the full path to that file, or nil.
@end defvar


@end defun


@defun semantic-find-nonterminal token &optional parent
Find the location of @var{TOKEN}.
@var{TOKEN} may be a stripped element, in which case @var{PARENT} specifies a
parent token that has position information.
Different behaviors are provided depending on the type of token.
For example, dependencies (includes) will seek out the file that is
depended on, and functions will move to the specified definition.
@end defun

@defun semantic-find-documentation token
Find documentation from @var{TOKEN} and return it as a clean string.
@var{TOKEN} might have @var{DOCUMENTATION} set in it already.  If not, there may be
some documentation in a comment preceding TOKEN's definition which we
can look for.  When appropriate, this can be overridden by a language specific
enhancement.
@end defun

@node Local Context, Making New Methods, Token Details, Override Methods
@subsection Local Context

@defun semantic-up-context &optional point
Move point up one context from @var{POINT}.
Return non-@code{nil} if there are no more context levels.
Overloaded functions using @code{up-context} take no parameters.
@end defun

@defun semantic-beginning-of-context &optional point
Move @var{POINT} to the beginning of the current context.
Return non-@code{nil} if there is no upper context.
The default behavior uses @dfn{semantic-up-context}.  It can
be overridden with @code{beginning-of-context}.
@end defun

@defun semantic-end-of-context &optional point
Move @var{POINT} to the end of the current context.
Return non-@code{nil} if there is no upper context.
Be default, this uses @dfn{semantic-up-context}, and assumes parenthetical
block delimiters.  This can be overridden with @code{end-of-context}.
@end defun

@defun semantic-get-local-variables &optional point
Get the local variables based on POINT's context.
Local variables are returned in Semantic token format.
Be default, this calculates the current bounds using context blocks
navigation, then uses the parser with @code{bovine-inner-scope} to
parse tokens at the beginning of the context.
This can be overridden with @code{get-local-variables}.
@end defun

@defun semantic-get-local-arguments &optional point
Get arguments (variables) from the current context at @var{POINT}.
Parameters are available if the point is in a function or method.
This function returns a list of tokens.  If the local token returns
just a list of strings, then this function will convert them to tokens.
Part of this behavior can be overridden with @code{get-local-arguments}.
@end defun

@defun semantic-get-all-local-variables &optional point
Get all local variables for this context, and parent contexts.
Local variables are returned in Semantic token format.
Be default, this gets local variables, and local arguments.
This can be overridden with @code{get-all-local-variables}.
Optional argument @var{POINT} is the location to start getting the variables from.
@end defun

These next set of functions handle local context parsing.   This means
looking at the code (locally) and navigating, and fetching information
such as a the type of the parameter the cursor may be typing in.

@defun semantic-end-of-command
Move to the end of the current command.
Be default, uses @code{semantic-command-separation-character}.
Override with @code{end-of-command}.
@end defun

@defun semantic-beginning-of-command
Move to the beginning of the current command.
Be default, users @code{semantic-command-separation-character}.
Override with @code{beginning-of-command}.
@end defun

@defun semantic-ctxt-current-symbol &optional point
Return the current symbol the cursor is on at @var{POINT} in a list.
This will include a list of type/field names when applicable.
This can be overridden using @code{ctxt-current-symbol}.
@end defun

@defun semantic-ctxt-current-assignment &optional point
Return the current assignment near the cursor at @var{POINT}.
Return a list as per @dfn{semantic-ctxt-current-symbol}.
Return @code{nil} if there is nothing relevant.
Override with @code{ctxt-current-assignment}.
@end defun

@defun semantic-ctxt-current-function &optional point
Return the current function the cursor is in at @var{POINT}.
The function returned is the one accepting the arguments that
the cursor is currently in.
This can be overridden with @code{ctxt-current-function}.
@end defun

@defun semantic-ctxt-current-argument &optional point
Return the current symbol the cursor is on at @var{POINT}.
Override with @code{ctxt-current-argument}.
@end defun

@defun semantic-ctxt-scoped-types &optional point
Return a list of type names currently in scope at @var{POINT}.
Override with @code{ctxt-scoped-types}.
@end defun

For details on using these functions to get more detailed information
about the current context: @xref{Context Analysis}.

@node Making New Methods,  , Local Context, Override Methods
@subsection Making New Methods

@node Parser Hooks, Example Programs, Override Methods, Programming
@section Parser Hooks

If you write a program that uses the stream of tokens in a persistent
display or database, it is necessary to know when tokens change so
that your displays can be updated.  This is especially important as
tokens can be replaced, changed, or deleted, and the associated
overlays will then throw errors when you try to use them.  Complete
integration with token changes can be achieved via several very
important hooks.

One interesting way to interact with the parser is to let it know that
changes you are going to make will not require re-parsing.

@defvar semantic-edits-are-safe
When non-@code{nil}, modifications do not require a reparse.
This prevents tokens from being marked dirty, and it
prevents top level edits from causing a cache check.
Use this when writing programs that could cause a full
reparse, but will not change the tag structure, such
as adding or updating top-level comments.
@end defvar

Next, it is sometimes useful to know what the current parsing state
is.  These function can let you know what level of re-parsing may be
needed.  Careful choices on when to reparse can make your program much
faster.

@defun semantic-bovine-toplevel-full-reparse-needed-p &optional checkcache
Return non-@code{nil} if the current buffer needs a full reparse.
Optional argument @var{CHECKCACHE} indicates if the cache check should be made.
@end defun

@defun semantic-bovine-toplevel-partial-reparse-needed-p &optional checkcache
Return non-@code{nil} if the current buffer needs a partial reparse.
This only returns non-@code{nil} if @dfn{semantic-bovine-toplevel-full-reparse-needed-p}
returns nil.
Optional argument @var{CHECKCACHE} indicates if the cache check should be made
when checking @dfn{semantic-bovine-toplevel-full-reparse-needed-p}.
@end defun

If you need very close interaction with the user's editing, then these
two hooks can be used to find out when a given tag is being changed.
These hooks could even be used to cut down on re-parsing if used correctly.

For all hooks, make sure you are careful to add it as a local hook if
you only want to effect a single buffer.  Setting it globally can
cause unwanted effects if your program is concerned with a single buffer.

@defvar semantic-dirty-token-hooks
Hooks run after when a token is marked as dirty (edited by the user).
The functions must take @var{TOKEN}, @var{START}, and @var{END} as a parameters.
This hook will only be called once when a token is first made dirty,
subsequent edits will not cause this to run a second time unless that
token is first cleaned.  Any token marked as dirty will
also be called with @code{semantic-clean-token-hooks}, unless a full
reparse is done instead.
@end defvar

@defvar semantic-clean-token-hooks
Hooks run after a token is marked as clean (re-parsed after user edits.)
The functions must take a @var{TOKEN} as a parameter.
Any token sent to this hook will have first been called with
@code{semantic-dirty-token-hooks}.  This hook is not called for tokens
marked dirty if the buffer is completely re-parsed.  In that case, use
@code{semantic-after-toplevel-cache-change-hook}.
@end defvar

@defvar semantic-change-hooks
Hooks run when semantic detects a change in a buffer.
Each hook function must take three arguments, identical to the
common hook @code{after-change-function}.
@end defvar

Lastly, if you just want to know when a buffer changes, use this hook.

@defvar semantic-after-toplevel-bovinate-hook
Hooks run after a toplevel token parse.
It is not run if the toplevel parse command is called, and buffer does
not need to be fully re-parsed.
This function is also called when the toplevel cache is flushed, and
the cache is emptied.
For language specific hooks, make sure you define this as a local hook.

This hook should not be used any more.
Use @code{semantic-after-toplevel-cache-change-hook} instead.
@end defvar

@defvar semantic-after-toplevel-cache-change-hook
Hooks run after the buffer token list has changed.
This list will change when a buffer is re-parsed, or when the token
list in a buffer is cleared.  It is *NOT* called if the current token
list partially re-parsed.

Hook functions must take one argument, which is the new list of
tokens associated with this buffer.

For language specific hooks, make sure you define this as a local hook.
@end defvar

@defvar semantic-after-partial-cache-change-hook
Hooks run after the buffer token list has been updated.
This list will change when the current token list has been partially
re-parsed.

Hook functions must take one argument, which is the list of tokens
updated among the ones associated with this buffer.

For language specific hooks, make sure you define this as a local hook.
@end defvar

@defvar semantic-before-toplevel-cache-flush-hook
Hooks run before the toplevel nonterminal cache is flushed.
For language specific hooks, make sure you define this as a local hook.
This hook is called before a corresponding
@code{semantic-after-toplevel-cache-change-hook} which is also called
during a flush when the cache is given a new value of nil.
@end defvar

@node Example Programs,  , Parser Hooks, Programming
@section Programming Examples

Here are some simple examples that use different aspects of the
semantic library APIs.  For fully functional example programs with
lots of comments, see the file @file{semantic-examples.el}.

@subsection Interactively querying for a token name

If you need a command that asks the user for a token name, you can
get full range completion using the query functions
@ref{Nonterminal Completion}.

@example
(interactive (list (semantic-read-symbol "Symbol: ")))
@end example

@subsection Finding a symbol in a buffer

If you have the name of a function or variable, and need to find its
location in a buffer, you need a search function.  There is a wide
range of searches you can perform @ref{Nonterminal Streams}.

@example
(semantic-find-nonterminal-by-name
 "some-name"
 (current-buffer)
 t    ;; look inside structures and classes for these symbols
 nil) ;; do not look inside header files.
@end example

@subsection Finding a symbol in a project

If you have the name of a function or variable, and need to find its
location somewhere in a project, you need to use the Semantic Database
@ref{semanticdb}.  There are many search functions similar to the
ones found in @ref{Nonterminal Streams}.

The Semantic Database is interesting in that the return structure is
not

@subsection Locating a token in a buffer

If you have a nonterminal token, or a list of them, you may want to
find their position in a buffer.

@example
(semanticdb-find-nonterminal-by-name
 "symbol"
 nil   ;; Defaults to the current project's database list.
 t     ;; Search inside types
 nil   ;; Do not search include files
 nil   ;; Only search files in the same mode (all C files)
 t     ;; When a token is found, make sure it is loaded in a buffer.
 )
@end example

Of interesting note above, semanticdb can find symbols in files that
are not loaded into an Emacs buffer.  These tokens do not have an
associated overlay, and the function @dfn{semantic-token-buffer} will
fail.

The last parameter's tells the search function to
@dfn{find-file-noselect} any file in which a matching token was
found.  This will allow you to merge all the tokens into a completion
list, or other flat list needed by most functions that use
association lists.

If you do not ask semanticdb to load those files, you will need to
explicitly request the database object (found in the @code{car} of
each sublist) get the file loaded.  It is useful to not auto find all
files if you don't need to jump to that token.

@subsection Converting a token into a human readable string.

A nonterminal token is a rather unpleasant Lisp structure when trying
to decipher what is going on.  As such, there is a wide range of
functions available that can convert a token into a human readable,
and colorful string @ref{Token->Text}.

If you program interfaces with lots of users, you will probably want
to have your program define a configurable variable that will let
users change the visible portion of your program.

@example
(defcustom my-summary-function 'semantic-uml-prototype-nonterminal
  "*Function to use when showing info about my token."
  :group 'my-program
  :type semantic-token->text-custom-list)
@end example

Note the special type provided by semantic.

Next, you can call this function to create a string.

@example
(funcall my-summary-function token
                             token-parent
                             t ; use color
                             )
@end example

In this case, @var{token-parent} is an optional argument.  In many
cases, parent is not used by the outputting function.  The parent
may be a struct or class that contains @var{token}, or nil for
top-level definitions.  In particular, C++ needs the parent to
correctly calculate the protection of each method.


@node Current Context, Tools, Programming, Top
@comment  node-name,  next,  previous,  up
@chapter Deriving the Current Context

This chapter deals with how to derive the current context, and also
how a language maintainer can get the current context API to work
with their language.

By default, the behavior will function in C like languages.  This
means languages with parenthetical blocks, and type dereferencing
which uses a similar form.

@menu
* Blocks::
* Local Variables::             Getting lists of local variables.
* Derived Context::             What goes at a given location?
* Context Analysis::            Analysis information about the local context.
@end menu

@node Blocks, Local Variables, Current Context, Current Context
@section Blocks and Navigation

Source code is typically built up of control structures, and blocks of
context, or lexical scope.  Semantic terms these lexical scopes as a
``context''.  The following functions can be used to navigate contexts.
Some of them are override functions.  Language authors can override
a subset of them to make them work for their language.

@defun semantic-up-context &optional point
Move point up one context from @var{POINT}.
Return non-@code{nil} if there are no more context levels.
Overloaded functions using @code{up-context} take no parameters.
@end defun

@defun semantic-beginning-of-context &optional point
Move @var{POINT} to the beginning of the current context.
Return non-@code{nil} if there is no upper context.
The default behavior uses @code{semantic-up-context}.  It can
be overridden with @code{beginning-of-context}.
@end defun

@defun semantic-end-of-context &optional point
Move @var{POINT} to the end of the current context.
Return non-@code{nil} if there is no upper context.
Be default, this uses @code{semantic-up-context}, and assumes parenthetical
block delimiters.  This can be overridden with @code{end-of-context}.
@end defun

These next set of functions can be used to navigate across commands.

@defun semantic-end-of-command
Move to the end of the current command.
Be default, uses @code{semantic-command-separation-character}.
Override with @code{end-of-command}.
@end defun

@defun semantic-beginning-of-command
Move to the beginning of the current command.
Be default, users @code{semantic-command-separation-character}.
Override with @code{beginning-of-command}.
@end defun

@node Local Variables, Derived Context, Blocks, Current Context
@section Deriving local variables

Within a given context, or block of code, local variables are often
defined.  These functions can be used to retrieve lists of locally
scoped variables.

@defun semantic-get-local-variables &optional point
Get the local variables based on POINT's context.
Local variables are returned in Semantic token format.
By default, this calculates the current bounds using context blocks
navigation, then uses the parser with @code{bovine-inner-scope} to
parse tokens at the beginning of the context.
This can be overridden with @code{get-local-variables}.
@end defun

@defun semantic-get-local-arguments &optional point
Get arguments (variables) from the current context at @var{POINT}.
Parameters are available if the point is in a function or method.
This function returns a list of tokens.  If the local token returns
just a list of strings, then this function will convert them to tokens.
Part of this behavior can be overridden with @code{get-local-arguments}.
@end defun

@defun semantic-get-all-local-variables &optional point
Get all local variables for this context, and parent contexts.
Local variables are returned in Semantic token format.
Be default, this gets local variables, and local arguments.
This can be overridden with @code{get-all-local-variables}.
Optional argument @var{POINT} is the location to start getting the variables from.
@end defun

@node Derived Context, Context Analysis, Local Variables, Current Context
@section Deriving the Current Context

While a context has already been used to describe blocks of code,
other context include more local details, such as the symbol the
cursor is on, or the fact we are assigning into some other variable.

These context deriving functions can be overridden to provide language
specific behavior.  By default, it assumes a C like language.

@defun semantic-ctxt-current-symbol &optional point
Return the current symbol the cursor is on at @var{POINT} in a list.
This will include a list of type/field names when applicable.
This can be overridden using @code{ctxt-current-symbol}.
@end defun

@defun semantic-ctxt-current-assignment &optional point
Return the current assignment near the cursor at @var{POINT}.
Return a list as per @code{semantic-ctxt-current-symbol}.
Return @code{nil} if there is nothing relevant.
Override with @code{ctxt-current-assignment}.
@end defun

@defun semantic-ctxt-current-function &optional point
Return the current symbol the cursor is on at @var{POINT}.
The function returned is the one accepting the arguments that
the cursor is currently in.
This can be overridden with `ctxt.current-function'.
@end defun

@defun semantic-ctxt-current-argument &optional point
Return the current symbol the cursor is on at @var{POINT}.
Override with @code{ctxt-current-argument}.
@end defun

@node Context Analysis,  , Derived Context, Current Context
@section Analysis of the current context

The context parsing API is used in a context analysis library.  This
library provides high level routines for scanning through token
databases to create lists of token associates.  At it's core is a set
of EIEIO classes defining a context.  The context contains information
about what was parsed at a given position, such as the strings there,
and they type of assignment.  The analysis library then searches the
databases to determine the types and names available.

Two high level functions which can be run interactively are:

*NOTE TO SELF: Add more here*

@deffn Command semantic-analyze-current-context position
Analyze the current context at @var{POSITION}.
If called interactively, display interesting information about @var{POSITION}
in a separate buffer.
Returns an object based on symbol @code{semantic-analyze-context}.
@end deffn


@deffn Command semantic-analyze-possible-completions point
Return a list of semantic tokens which are possible completions.
Analysis is done at @var{POINT}.
@end deffn

@node Tools, Index, Current Context, Top
@comment  node-name,  next,  previous,  up
@chapter Tools

Several tools come with Semantic which would not be possible without it.
In general, these tools will work with any language supported by
Semantic.

@menu
* speedbar::                    How to use Semantic speedbar support
* imenu::                       Special support for Imenu.
* semanticdb::                  Cache your parsed buffers between sessions.
* senator::                     The Semantic Navigator
* analyzer::                    Analyze local context
* class browser::               Class hierarchy browser
* document::                    Document generation functions
* charts::                      Charting code statistics
* minor modes::                 Useful minor modes
@end menu

@node speedbar, imenu, Tools, Tools
@comment  node-name,  next,  previous,  up
@section Speedbar

Speedbar supports the display of tags through the Semantic parser.  To
use this utility, add a line like this to your @file{.emacs} file:

@example
(add-hook 'speedbar-load-hook (lambda () (require 'semantic-sb)))
@end example

or you can simply add:

@example
(require 'semantic-sb)
@end example

Once installed, speedbar will use semantic to find tokens, and will
display them appropriately.  Tags from semantic will have additional
details which can be seen, such as return type, or arguments to
functions.

If you use @file{semantic-load.el}, you do not need to add the above
lines in your .emacs file.

Two additional speedbar modes are described in @ref{Speedbar Analysis}, and
@ref{class browser}.

@node imenu, semanticdb, speedbar, Tools
@comment  node-name,  next,  previous,  up
@section Imenu support

There is special support for creating Imenu entries using semantic.
This is a highly customizable tool which can create specialized menu
systems for navigating your source file.

By default, each language that wants special imenu support will set
itself up for it.  To setup imenu for your buffers, use this command
in your @file{.emacs} file:

@example
(add-hook 'semantic-init-hooks (lambda ()
                               	   (imenu-add-to-menubar "TOKENS")))
@end example

Also supported is @dfn{which-func-mode}.  This usually uses imenu
tags to show the current function.  The semantic support for this
function uses overlays, which is much faster.

If you use @file{semantic-load.el}, you do not need to add the above
lines in your .emacs file.

You can customize imenu with the following options:

@deffn Option semantic-imenu-summary-function
Function to use when creating items in Imenu.
Some useful functions are:
@dfn{semantic-abbreviate-nonterminal}
@dfn{semantic-summarize-nonterminal}
@dfn{semantic-prototype-nonterminal}
@end deffn

@deffn Option semantic-imenu-bucketize-file
Non-@code{nil} if tokens in a file are to be grouped into buckets.
@end deffn

@deffn Option semantic-imenu-buckets-to-submenu
Non-@code{nil} if buckets of tokens are to be turned into submenus.
This option is ignored if @code{semantic-imenu-bucketize-file} is nil.
@end deffn

@deffn Option semantic-imenu-expand-type-parts
Non-@code{nil} if types should have submenus with parts in it.
@end deffn

@deffn Option semantic-imenu-bucketize-type-parts
Non-@code{nil} if elements of a type should be placed grouped into buckets.
@code{Nil} means to keep them in the same order.
Overridden to @code{nil} if @code{semantic-imenu-bucketize-file} is nil.
@end deffn

@deffn Option semantic-imenu-sort-bucket-function
Function to use when sorting tags in the buckets of functions.
@end deffn

@deffn Option semantic-imenu-index-directory
Non @code{nil} to index the entire directory for tags.
Doesn't actually parse the entire directory, but displays tags for all files
currently listed in the current Semantic database.
This variable has no meaning if semanticdb is not active.
@end deffn

@deffn Option semantic-imenu-auto-rebuild-directory-indexes
If non-@code{nil} automatically rebuild directory index imenus.
That is when a directory index imenu is updated, automatically rebuild
other buffer local ones based on the same semanticdb.
@end deffn

When adding support to a language, this variable may need to be set:

@defvar semantic-imenu-expandable-token
Tokens of this token type will be given submenu with children.
By default, a @code{type} has interesting children.  In Texinfo, however,
a @code{section} has interesting children.
@end defvar


@node semanticdb, senator, imenu, Tools
@comment  node-name,  next,  previous,  up
@section Semantic Database

Semanticdb is a utility which tracks your parsed files, and saves the
parsed information to files.  When you reload your source files,
semanticdb automatically associates the file with the cached copy,
saving time by not re-parsing your buffer.

Semanticdb also provides an API for programs to use.  These functions
will return token information without loading the source file into
memory by checking the disk cache.

To use semanticdb, add the following to your @file{.emacs} file:

@example
(require 'semanticdb)
(global-semanticdb-minor-mode 1)
@end example

If you have a tool which optionally uses the semantic database, it may
be important to track if the database mode is turned on or off.

@deffn Option semanticdb-mode-hooks
Hooks run whenever @dfn{global-semanticdb-minor-mode} is run.
Use @dfn{semanticdb-minor-mode-p} to determine if the mode has been turned
on or off.
@end deffn

@deffn Option semanticdb-persistent-path
List of valid paths that semanticdb will cache tokens to.
When @dfn{global-semanticdb-minor-mode} is active, token lists will
be saved to disk when Emacs exits.  Not all directories will have
tokens that should be saved.
The value should be a list of valid paths.  @var{A} path can be a string,
indicating a directory in which to save a variable.  An element in the
list can also be a symbol.  Valid symbols are @code{never}, which will
disable any saving anywhere, @code{always}, which enables saving
everywhere, or @code{project}, which enables saving in any directory that
passes a list of predicates in @code{semantic-project-predicates}.
@end deffn

@deffn Option semanticdb-project-roots
List of directories, where each directory is the root of some project.
All subdirectories of a root project are considered a part of one project.
Values in this string can be overridden by project management programs
via the @code{semanticdb-project-root-functions} variable.
@end deffn

The important difference between these two is that you may put just
``~'' in @code{semanticdb-persistent-path}, but you may put individual
project directories into @code{semanticdb-project-roots} so that
different database lists don't get cross referenced incorrectly.

@subsection Searching

You can search for tokens in the database using the following
functions.  It is important to note that database search functions do
not return a plain list of tokens.  This is because some tokens may
not be loaded in a buffer, which means that the found token would not
have an overlay, and no way to determine where it came from.

As such, all search functions return a list of the form:

@example
   ( (DATABASE TOKEN1 TOKEN2 ...) (DATABASE2 TOKEN3 TOKEN4 ...) ...)
@end example

@defun semanticdb-find-nonterminal-by-function function &optional databases search-parts search-includes diff-mode find-file-match
Find all occurrences of nonterminals which match @var{FUNCTION}.
Search in all @var{DATABASES}.  If @var{DATABASES} is @code{nil}, search a range of
associated databases.
When @var{SEARCH-PARTS} is non-@code{nil} the search will include children of tokens.
When @var{SEARCH-INCLUDES} is non-@code{nil}, the search will include dependency files.
When @var{DIFF-MODE} is non-@code{nil}, search databases which are of a different mode.
@var{A} Mode is the @code{major-mode} that file was in when it was last parsed.
When @var{FIND-@var{FILE}-MATCH} is non-@code{nil}, the make sure any found token's file is
in an Emacs buffer.
@end defun

@defun semanticdb-find-nonterminal-by-name name &optional databases search-parts search-includes diff-mode find-file-match
Find all occurrences of nonterminals with name @var{NAME} in databases.
See @dfn{semanticdb-find-nonterminal-by-function} for details on @var{DATABASES},
@var{SEARCH-PARTS}, @var{SEARCH-INCLUDES}, @var{DIFF-MODE}, and @var{FIND-@var{FILE}-MATCH}.
@end defun

@defun semanticdb-find-nonterminal-by-name-regexp regex &optional databases search-parts search-includes diff-mode find-file-match
Find all occurrences of nonterminals with name matching @var{REGEX} in databases.
See @dfn{semanticdb-find-nonterminal-by-function} for details on @var{DATABASES},
@var{SEARCH-PARTS}, @var{SEARCH-INCLUDES} @var{DIFF-MODE}, and @var{FIND-@var{FILE}-MATCH}.
@end defun

@defun semanticdb-find-nonterminal-by-type type &optional databases search-parts search-includes diff-mode find-file-match
Find all nonterminals with a type of @var{TYPE} in databases.
See @dfn{semanticdb-find-nonterminal-by-function} for details on @var{DATABASES},
@var{SEARCH-PARTS}, @var{SEARCH-INCLUDES} @var{DIFF-MODE}, and @var{FIND-@var{FILE}-MATCH}.
@end defun

@defun semanticdb-find-nonterminal-by-property property value &optional databases search-parts search-includes diff-mode find-file-match
Find all nonterminals with a @var{PROPERTY} equal to @var{VALUE} in databases.
See @dfn{semanticdb-find-nonterminal-by-function} for details on @var{DATABASES},
@var{SEARCH-PARTS}, @var{SEARCH-INCLUDES} @var{DIFF-MODE}, and @var{FIND-@var{FILE}-MATCH}.
Return a list ((@var{DB-TABLE} . @var{TOKEN-LIST}) ...).
@end defun

@defun semanticdb-find-nonterminal-by-extra-spec spec &optional databases search-parts search-includes diff-mode find-file-match
Find all nonterminals with a @var{SPEC} in databases.
See @dfn{semanticdb-find-nonterminal-by-function} for details on @var{DATABASES},
@var{SEARCH-PARTS}, @var{SEARCH-INCLUDES} @var{DIFF-MODE}, and @var{FIND-@var{FILE}-MATCH}.
Return a list ((@var{DB-TABLE} . @var{TOKEN-LIST}) ...).
@end defun

@defun semanticdb-find-nonterminal-by-extra-spec-value spec value &optional databases search-parts search-includes diff-mode find-file-match
Find all nonterminals with a @var{SPEC} equal to @var{VALUE} in databases.
See @dfn{semanticdb-find-nonterminal-by-function} for details on @var{DATABASES},
@var{SEARCH-PARTS}, @var{SEARCH-INCLUDES} @var{DIFF-MODE}, and @var{FIND-@var{FILE}-MATCH}.
Return a list ((@var{DB-TABLE} . @var{TOKEN-LIST}) ...).
@end defun

@defun semanticdb-file-stream file
Return a list of tokens belonging to @var{FILE}.
If file has database tokens available in the database, return them.
If file does not have tokens available, then load the file, and create them.
@end defun

@node senator, analyzer, semanticdb, Tools
@comment  node-name,  next,  previous,  up
@section Semantic Navigator

Senator stands for SEmantic NAvigaTOR and was written by David Ponce.

This library defines commands and a minor mode to navigate between
semantic language tokens in the current buffer.

@subsection Commands

The following user level commands are provided by Senator.

@subsubsection Navigation

@deffn Command senator-next-token
Move to the next token in the current buffer.
@end deffn

@deffn Command senator-previous-token
Move to the previous token in the current buffer.
@end deffn

@deffn Command senator-jump sym
Jump to the semantic symbol @var{SYM}.
If called interactively and a prefix argument is supplied jump in the
local type's context (see function @dfn{senator-current-type-context}).
@end deffn

@subsubsection Searching

Searching using senator mode restricts the search only to the
definition text, such as the name of the functions or variables in a
given buffer.

@deffn Command senator-isearch-toggle-semantic-mode
Toggles semantic search in isearch mode.  When semantic search is
enabled, isearch is restricted to token names.
@end deffn

@deffn Command senator-search-forward string
@deffnx Command senator-search-backward string
Search forward and backward for a token matching @var{string}.
@end deffn

@deffn Command re-search-forward regex
@deffnx Command re-search-backward regex
Search forward and backward for a token matching the regular expression
@var{regex}.
@end deffn

@deffn Command word-search-forward word
@deffnx word-search-backward word
Search forward and backward for a token whose name matches @var{word}.
@end deffn

@subsubsection Completion

Completion in senator scans all known definitions in the local file,
and uses that information to provide the completion.

@deffn Command senator-complete-symbol
Complete the current symbol under point.
@end deffn

@deffn Command senator-completion-menu-keyboard-popup
Popup a completion menu for the symbol at point.
@end deffn

@subsubsection Copy/Paste

Token Copy/Paste is a high level form of the typical copy yank used by
Emacs.  Copying a token saves the meta-information related to the
function or item the cursor is currently in.  When that information is
yanked into a new buffer, the form of the text created is based on the
current status of the programming buffer.

For example, pasting a function into a different file results in a
function call template to be inserted.  In a Texinfo file, a @@deffn
is created with documentation for that function or command.

@deffn Command senator-copy-token
Take the current token, and place it in the token ring.
@end deffn

@deffn Command senator-kill-token
Take the current token, place it in the token ring, and kill it.
Killing the token removes the text for that token, and places it into
the kill ring.  Retrieve that text with @code{yank}.
@end deffn

@deffn Command senator-yank-token
Yank a token from the token ring.
The form the token takes is different depending on where it is being
yanked to.
@end deffn

@deffn Command senator-copy-token-to-register register &optional kill-flag
Copy the current token into @var{REGISTER}.
Optional argument @var{KILL-FLAG} will delete the text of the token to the
kill ring.
@end deffn

For programmers, to provide specialized pasting, created an override
function for @code{insert-foreign-token} (see @xref{Settings}.)

@subsubsection Minor Mode

@deffn Command senator-minor-mode
Toggle the SEmantic NAvigaTOR key bindings in the current buffer.

The following default key bindings are provided when semantic minor
mode is enabled:


@table @key
@item C-c , n
senator-next-token
@item C-c , p
senator-previous-token
@item C-c , i
senator-isearch-toggle-semantic-mode
@item C-c , j
senator-jump
@item C-c , TAB
senator-complete-symbol
@item C-c , SPC
senator-completion-menu-keyboard-popup
@item C-c , C-y
senator-yank-token
@item C-c , C-w
senator-kill-token
@item C-c , M-w
senator-copy-token
@item C-c ,
@end table
@end deffn

@subsection Customization

To enable the Senator keymap in all modes that support semantic parsing,
use this:

@example
(add-hook 'semantic-init-hooks 'senator-minor-mode)
@end example

To customize navigation around different types of tokens, use the
following variables:

@deffn Option senator-step-at-token-ids
List of token identifiers where to step.
Token identifier is symbol @code{'variable}, @code{'function}, @code{'type}, or other.  If
@code{nil} navigation steps at any token found.  This is a buffer local
variable.  It can be set in a mode hook to get a specific language
navigation.
@end deffn

@deffn Option senator-step-at-start-end-token-ids
List of token identifiers where to step at start and end.
Token identifier is symbol @code{'variable}, @code{'function}, @code{'type}, or other.  If
@code{nil} navigation only step at beginning of tokens.  If @code{t} step at start
and end of any token where it is allowed to step.  Also, stepping at
start and end of a token prevent stepping inside its children.  This
is a buffer local variable.  It can be set in a mode hook to get a
specific language navigation.
@end deffn

To have a mode specific customization, do something like this in a hook:

@example
(add-hook 'mode-hook
          (lambda ()
            (setq senator-step-at-token-ids '(function variable))
            (setq senator-step-at-start-end-token-ids '(function))
            ))
@end example

This will cause navigation and search commands to stop only between
functions and variables, and to step at start and end of functions
only.

@subsection Contact information for Senator

Any comments, suggestions, bug reports or upgrade requests are
welcome.  Please send them to David Ponce at david@@dponce.com

@node analyzer, class browser, senator, Tools
@comment  node-name,  next,  previous,  up
@section Semantic Analyzer

The semantic analyzer is a library tool that performs context
analysis and can derive useful information.

@deffn Command semantic-analyze-current-context position
Analyze the current context at @var{POSITION}.
If called interactively, display interesting information about @var{POSITION}
in a separate buffer.
Returns an object based on symbol @dfn{semantic-analyze-context}.
@end deffn

While this can be used as a command, it is mostly useful that way
while debugging the analyzer, or tools using the return value.  Use
the Emacs command @dfn{describe-class} to learn more about using
@code{semantic-analyze-context}.

Another command that uses the analyzer context derives a completion
list.

@deffn Command semantic-analyze-possible-completions context
Return a list of semantic tokens which are possible completions.
@var{CONTEXT} is either a position (such as point), or a pre-calculated
context.  Passing in a context is useful if the caller also needs
to access parts of the analysis.

Completions run through the following filters:

@table @asis
@item Elements currently in scope
@item Constants currently in scope
@item Elements match the @code{:prefix} in the @var{CONTEXT}.
@item Type of the completion matches the type of the context.
@end table

Context type matching can identify the following:

@table @asis
@item No specific type
@item Assignment into a variable of some type.
@item Argument to a function with type constraints.
@end table

When called interactively, this function displays the list of
possible completions.  This is useful for debugging.
@end deffn

@menu
* Smart Completion ::           Commands for smart completion.
* Speedbar Analysis ::          Display an analysis in speedbar.
@end menu

@node Smart Completion, Speedbar Analysis, analyzer, analyzer
@comment  node-name,  next,  previous,  up
@subsection Smart Completion

The file @file{semantic-ia.el} contains two commands for performing
smart completion using the analysis library.  Analysis to calculate
these completions are done through the analyzer and completion
mechanism.  These functions just provide commands that can be bound
to key bindings.

@deffn Command semantic-ia-complete-symbol point
Complete the current symbol at @var{POINT}.
Completion options are calculated with
@dfn{semantic-analyze-possible-completions}.
@end deffn

@deffn Command semantic-ia-complete-symbol-menu point
Complete the current symbol via a menu based at @var{POINT}.
Completion options are calculated with
@dfn{semantic-analyze-possible-completions}.
@end deffn

@node Speedbar Analysis,  , Smart Completion, analyzer
@comment  node-name,  next,  previous,  up
@subsection Speedbar Analysis

The Analyzer output can be used through a speedbar interface.
This interface lists details about the analysis, such as the current
function, local arguments and variables, details on the prefix (the
symbol the cursor is on), and a list of all possible completions.
Completions are specified in
@dfn{semantic-analyze-possible-completions}
@ref{analyzer}.

Each entry can be jumped to by clicking on the name.  For strongly
typed languages, this means you will jump to the definition of the
variable, slot, or type definition.

In addition each entry has an <i> button.  Clicking on this will
display a summary of everything that is known about the variable or
type displayed on that line.

If you click on the name of a variable in the ``Completions'' menu,
then the text that was recently analyzed will be replaced with the
name of the token that was clicked on in speedbar.

@deffn Command semantic-speedbar-analysis
Start Speedbar in semantic analysis mode.
The analyzer displays information about the current context, plus a smart
list of possible completions.
@end deffn

You can also enter speedbar analyzer mode by selecting ``Analyze''
from the ``Display'' menu item on speedbar's menu.

@node class browser, document, analyzer, Tools
@comment  node-name,  next,  previous,  up
@section Class Browser

The semantic class browser is a library that can covert a project
group of files into an EIEIO based structure that contains links
between structures so that the inheritance links between semantic
tokens can be easily navigated.

The core to this library is one function in @file{semantic-cb.el}.

@defun semantic-cb-new-class-browser
Create an object representing this project's organization.
The object returned is of type @dfn{semantic-cb-project}, which contains
the slot `:types', a list of all top-level types.  Each element is a
class of type @dfn{semantic-cb-token}, or @dfn{semantic-cb-type}.
@end defun

Use the Emacs function `describe-class' to learn more about these
classes.

You can access the class inheritance structure through a speedbar
interface.  You can choose the ``Class Browser'' option from
Speedbar's ``Display'' menu item, or use the following command:

@deffn Command semantic-cb-speedbar-mode
Bring speedbar up, and put it into Class Browser mode.
This will use the Class Browser logic applied to the current Semantic
project database to build the available relations.  The structure of
the class hierarchy can then be navigated using traditional speedbar
interactions.
@end deffn

@node document, charts, class browser, Tools
@comment  node-name,  next,  previous,  up
@section Document generation

The document program uses semantic token streams to aid in the
creation of texinfo documentation.
For example, the following is a code fragment from @file{document.el}
that comes with semantic:

@lisp
(defun document (&optional resetfile)
  "Document the function or variable the cursor is in.
Optional argument RESETFILE is provided w/ universal argument.
When non-nil, query for a new documentation file."
  ...
  )
@end lisp

While visiting @file{document.el}, put the cursor somewhere within the
function shown above. Then type @kbd{M-x document}.
After asking for the texinfo file name, which in this case is
@file{semantic.texi}, this will update the texinfo
documentation of the @code{document} function in that file.
The result is that the following texinfo text will be either created
or updated in @file{semantic.texi} file:

@example
@@deffn Command document &optional resetfile
Document the function or variable the cursor is in.
Optional argument @@var@{RESETFILE@} is provided w/ universal argument.
When non-@@code@{nil@}, query for a new documentation file.
@@end deffn
@end example

Note that the function name, arguments and documentation string
is put in the right place.
Within the doc-string, the function arguments are marked with
the @@var command and the @code{nil} code fragment is marked with
@@code command.
This example provides just a glimpse of what is possible with the
syntactic information provided by semantic.

The main entry point for the
documentation generator are the following commands:

@deffn Command document &optional resetfile
Document the function or variable the cursor is in.
Optional argument @var{RESETFILE} is provided w/ universal argument.
When non-@code{nil}, query for a new documentation file.
@end deffn

@deffn Command document-inline
Document the current function with an inline comment.
@end deffn

@deffn Command document-insert-defun-comment nonterm buffer
Insert mode-comment documentation about @var{NONTERM} from @var{BUFFER}.
@end deffn

@deffn Command document-insert-new-file-header header
Insert a new header file into this buffer.  Add reference to @var{HEADER}.
Used by @code{prototype} if this file doesn't have an introductory comment.
@end deffn

In addition to these base documentation commands, the texinfo semantic
parser includes a two convenience functions when working directly with
texinfo files.

@deffn Command semantic-texi-update-doc &optional token
Update the documentation for @var{TOKEN}.
If the current buffer is a texinfo file, then find the source doc, and
update it.  If the current buffer is a source file, then get the
documentation for this item, find the existing doc in the associated
manual, and update that.
@end deffn

@deffn Command semantic-texi-goto-source &optional token
Jump to the source for the definition in the texinfo file @var{TOKEN}.
If @var{TOKEN} is @code{nil}, it is derived from the deffn under @var{POINT}.
@end deffn

@node charts, minor modes, document, Tools
@comment  node-name,  next,  previous,  up
@section Charting Commands

Some commands to draw charts of statistics generated from parsing:

@deffn Command semantic-chart-nonterminals-by-token &optional buffer-or-stream
Create a bar chart representing the number of nonterminals for a token.
Each bar represents how many toplevel nonterminal in @var{BUFFER-OR-STREAM}
exist with a given token type.  See `semantic-symbol->name-assoc-list'
for tokens which will be charted.
@end deffn

@deffn Command semantic-chart-database-size &optional buffer-or-stream
Create a bar chart representing the size of each file in semanticdb.
Each bar represents how many toplevel nonterminals in @var{BUFFER-OR-STREAM}
exist in each database entry.
@end deffn

@deffn Command semantic-chart-nonterminal-complexity-token &optional symbol buffer-or-stream
Create a bar chart representing the complexity of some tokens.
Complexity is calculated for tokens with a token of @var{SYMBOL}.  Each bar
represents the complexity of some nonterminal in @var{BUFFER-OR-STREAM}.
Only the most complex items are charted.
@end deffn


@node minor modes,  , charts, Tools
@comment  node-name,  next,  previous,  up
@section Minor Modes

@deffn Command semantic-show-dirty-mode &optional arg
Minor mode for highlighting dirty tokens.
With prefix argument @var{ARG}, turn on if positive, otherwise off.  The
minor mode can be turned on only if semantic feature is available and
the current buffer was set up for parsing.  Return non-@code{nil} if the
minor mode is enabled.
@end deffn

@deffn Command global-semantic-show-dirty-mode &optional arg
Toggle global use of @code{semantic-show-dirty-mode}.
If @var{ARG} is positive, enable, if it is negative, disable.
If @var{ARG} is @code{nil}, then toggle.
@end deffn

@deffn Option semantic-dirty-token-face
Face used to show dirty tokens in @code{semantic-show-dirty-token-mode}.
@end deffn

@deffn Command semantic-show-unmatched-syntax-mode &optional arg
Minor mode to highlight unmatched-syntax tokens.
With prefix argument @var{ARG}, turn on if positive, otherwise off.  The
minor mode can be turned on only if semantic feature is available and
the current buffer was set up for parsing.  Return non-@code{nil} if the
minor mode is enabled.
@end deffn

@deffn Command global-semantic-show-unmatched-syntax-mode &optional arg
Toggle global use of @dfn{semantic-show-unmatched-syntax-mode}.
If @var{ARG} is positive, enable, if it is negative, disable.
If @var{ARG} is @code{nil}, then toggle.
@end deffn

@deffn Option semantic-unmatched-syntax-face
Face used to show unmatched-syntax in.
The face is used in  @dfn{semantic-show-unmatched-syntax-mode}.
@end deffn

@deffn Command global-semantic-auto-parse-mode &optional arg
Toggle global use of @dfn{semantic-auto-parse-mode}.
If @var{ARG} is positive, enable, if it is negative, disable.
If @var{ARG} is @code{nil}, then toggle.
@end deffn

@deffn Command semantic-auto-parse-mode &optional arg
Minor mode to auto parse buffer following changes.
With prefix argument @var{ARG}, turn on if positive, otherwise off.  The
minor mode can be turned on only if semantic feature is available and
the current buffer was set up for parsing.  Return non-@code{nil} if the
minor mode is enabled.
@end deffn

@deffn Option semantic-auto-parse-no-working-message
Non-@code{nil} disable display of working message during parse.
@end deffn

@deffn Option semantic-auto-parse-idle-time
Time in seconds of idle time before auto-reparse.
This time should be short enough to ensure that auto-parse will be
run as soon as Emacs is idle.
@end deffn

@deffn Option semantic-auto-parse-max-buffer-size
Maximum size in bytes of buffers automatically re-parsed.
If this value is less than or equal to @var{0} buffers are automatically
re-parsed regardless of their size.
@end deffn

@node Index,  , Tools, Top
@comment  node-name,  next,  previous,  up
@chapter Index

@unnumbered Function Index
@printindex fn

@bye

@c LocalWords: alist API APIs arg argc args argv asis assoc autoload
@c LocalWords: backquote bnf bovinate bovinates
@c LocalWords: bovinating bovination bovinator bucketize
@c LocalWords: cb cdr charquote checkcache cindex CLOS
@c LocalWords: concat concocting const constantness ctxt Decl defcustom
@c LocalWords: deffn deffnx defun defvar destructor's dfn diff dir
@c LocalWords: doc docstring EDE EIEIO elisp emacsman emph enum
@c LocalWords: eq Exp EXPANDFULL expresssion fn foo func funcall
@c LocalWords: ia ids iff ifinfo imenu imenus init int isearch itemx java kbd
@c LocalWords: keymap keywordtable lang languagemode lexer lexing Ludlam
@c LocalWords: menubar metaparent metaparents min minibuffer Misc mode's
@c LocalWords: multitable NAvigaTOR noindent nomedian nonterm noselect
@c LocalWords: nosnarf obarray OLE OO outputfile paren parsetable POINT's
@c LocalWords: popup positionalonly positiononly positionormarker pre
@c LocalWords: printf printindex Programmatically pt punctuations quotemode
@c LocalWords: ref regex regexp Regexps reparse resetfile samp sb
@c LocalWords: scopestart SEmantic semanticdb setfilename setq
@c LocalWords: settitle setupfunction sexp sp SPC speedbar speedbar's
@c LocalWords: streamorbuffer struct subalist submenu submenus
@c LocalWords: subsubsection sw sym texi texinfo titlefont titlepage
@c LocalWords: tok TOKEN's toplevel typemodifiers uml unset untar
@c LocalWords: uref usedb var vskip xref yak
Tip: Filter by directory path e.g. /media app.js to search for public/media/app.js.
Tip: Use camelCasing e.g. ProjME to search for ProjectModifiedEvent.java.
Tip: Filter by extension type e.g. /repo .js to search for all .js files in the /repo directory.
Tip: Separate your search with spaces e.g. /ssh pom.xml to search for src/ssh/pom.xml.
Tip: Use ↑ and ↓ arrow keys to navigate and return to view the file.
Tip: You can also navigate files with Ctrl+j (next) and Ctrl+k (previous) and view the file with Ctrl+o.
Tip: You can also navigate files with Alt+j (next) and Alt+k (previous) and view the file with Alt+o.