Introduction

SkAT (Skeletons and Application Templates) is a set of tools that allows users to create template languages, template-based code generators and aspect weavers. It builds upon the model of "invasive software composition (ISC)" for fragment composition and uses "reference attribute grammars (RAGs)" to specify languages and composition systems. The basic implementation is based on Java and the JastAdd metacompiler for RAGs.

Invasive Software Composition (ISC)

ISC is a generic approach for fragment-based composition systems where a fragment is partial or underspecified piece of a program or model. ISC is a language-aware approach, i.e., it is syntax-safe according to the grammar or language metamodel. Typical applications of ISC comprise:

• Syntax-safe code generators and template engines (like C++ templates)
• Syntax-safe pre-processors (like a syntax-safe CPP)
• Model composition (like Reuseware)
• Aspect-oriented programming (like AspectJ, JBoss AOP)
• Variability Modelling (Feature Modelling)

A fragment component in ISC can be an arbitrary syntactic concept which is defined in the grammar of the component language or its metamodel, for example

• a method declaration,
• a field declaration,
• an expression,
• a whole class declaration.

A fragment component provides a set of compositional points which make up its fragment interface. A slot is an explicitly declared variation point in the fragment having a syntactic type, determined by the slots position in the fragment. In contrast to slots, hooks are intentional (implicit) compositional points of a fragment induced by the structure of the fragment language. Typically, implicit hooks occur where language concepts have an unbound number of enclosed elements, like the list of statements in a method block or the list of member declarations of a class.

Slots can be replaced by a compatible fragment component via the bind composition operator. Given a point and a fragment, bind first checks if the fragment is allowed to safely replace the slot and then performs the rewrite of the program. After the composition, the composed fragment is guaranteed to conform to the grammar of the fragment language. A typical application of the bind composer is the instantiation of a template class with type parameters. Hooks can be extended via the extend composer which performs the syntactic checking and the rewrite of the fragment.

A composition program (also composition recipe) is a collection of bind and extend composition steps intermixed with or without additional conditional and recursive constructs to control composition. A composition program is written in a composition language that can be declarative, imperative or domain-specific. Examples for declarative composition languages include C++ templates and aspect languages like AspectJ. Imperative composition languages can be a metaprogram written, e.g., in Java or an imperative template expansion language like PHP. Frequently, domain-specific composition languages are declarative (e.g, the composition diagrams of Reuseware, or feature models).

SkAT's Extensions to ISC

SkAT implements the basic model of invasive software composition as described above and in the corresponding blue book by Uwe Aßmann. However, it extends this model in three major directions:

• Well-formed ISC: adds static semantics and constraint checking capabilities
• Minimal ISC: provides a minimalistic approach to ISC by using partial component models
• Scalable ISC: a process to develop ISC systems starting with low implementation efforts extending them further using agile software-development approaches

The following figure characterizes the relations of these extensions and their relations to the original ISC model. Moreover, the following paragraphs provide more details on the extensions.

Well-Formed Invasive Software Composition

Well-formed ISC extends the standard model of ISC with a support for constraint checking during composition. These constraints can be based on context-sensitive properties of the underlying language such as visibility of variable names or the language's type system. For example, a well-formed composition system can ensure that variables used in a fragment component are actually visible at the point where the fragment is inserted and that they have an adequate type.

This way, errors can be discovered early at composition time (or template-expansion time) instead of compile time. Moreover, the system can issue cause-related error messages instead of the cryptic messages by the compiler, as shown in the figure below.

Another issue in invasive software composition (and in rewrite systems in general) is the order in which single composition steps are executed matters and may cause composition conflicts. Below, the scenario is extended to extend a method hook with two fragments.

Depending on their insertion order, the resulting program is different. As the log statement may be inserted before or after the write access to the field, it prints different results in these cases. While for logging this might be unproblematic, this causes severe problems if this was a message to a critical part of a software system which reacts differently depending on the value reported.

As a tool for handling these situations, well-formed ISC comprises a set of parametrizable composition strategies to make composition deterministic:

• Operator-determined composition interprets a collection of composer-call declarations (e.g., {bind(..), extend(..), bind(..),extract(..)}) as a set of rewrite rules that are applied by a fix-point iteration. Most importantly, their actual application order is determined alternatively by order of definition, kind of operator or an attribute-grammar-based analysis.

• Point-determined composition. Composer-call declarations are interpreted as a set of advices in the spirit of aspect-oriented programming with compositional points as static joinpoints. The order of application then is determined by a depth-first or breadth-first traversal over the fragment components involved. At each compositional point (hook, slot, or rudiment), composer-call declarations are looked up and applied depending on their order of occurrence or composer type.

• Attribute-determined composition. Composers are applied on demand and interleaved with attribute evaluation. Consequently, the composition results depend on the attribute-dependency graph induced by the attribute grammar of the fragment-language. Since the dependency graphs are not obvious in most cases, the application order of composer calls at composition time is also not obvious.

• Imperative mode. Composers are applied immediately via a composition API. This way, the composition is under full control of the calling application including application order and control flow.

Minimal Invasive Software Composition

Minimal ISC downgrades the standard model of ISC to string-based composition. That means, that fragment component models are not required to be defined with respect to a fragment-language grammar. While this drops all guarantees on the composition result, initial system implementation and prototyping of component models. Beyond plain infancy strings, minimal ISC uses so called island grammars to enable partially safe composition by recognizing some constructs as islands while other remain unrecognized in the stream of characters (i.e., the water). This way, composition becomes possible for any fragment in the fragment language while not everything must be specified. This is a clear advantage over the mini-language approach where only a small subset of a fragment language can be treated.

Scalable Invasive Software Composition

Scalable ISC combines minimal, classic and well-formed ISC by leveraging the power of extensible language specification mechanisms such as attribute grammars and island grammars. Starting of with a minimal component model, a full-fledged model is developed step-by-step by adding new component-model specifications and grammars. The following figure illustrates this:

Starting off with a minimal component model that only supports slots (iteration 1), during iterations 2 to n this model is refined with new island constructs supporting more constructs of the fragment language as well as slots and hooks therein. In iterations n+1 to m, some of the semantics of the fragment language or other constraints may already be considered in fragment contracts. In iteration m+1, the system already fully supports the fragment language, while in the iterations up to m+k, more constraints and semantic rules such as the type system of the fragment language are taken into account, making the composition safe.

Note that since with each iteration a fully operational system is obtained, scalable ISC is compatible with agile software-development methods. Furthermore, the development may start at any stage i>0 and end at any stage j (j>i-1).

SkAT's Architecture and Code Generation Process

SkAT has a layered architecture as sketched in the figure below:

As already mentioned, at the lowest layer, SkAT relies on the JastAdd system. In doing so, layers above provide JastAdd attribute grammar specifications, Java classes and parser grammars (not shown). In the composition frameworks & languages layer, SkAT/Core, SkAT/Full and SkAT/Minimal provide the basic infrastructure for composition environments that is used by the specific composition systems (how this is done is explained in more detail in the code-generation-process paragraph). At the same layer, specific fragment-language specifications reside (e.g., JastAddJ a Java compiler based on JastAdd attribute grammars). The two layers above contain composition tools. In the functional composition systems layer, basic composition systems are described, which provide component models and a composition API so that composition libraries can be generated and used in other applications (the specific systems mentioned here, STpL and SkAT4J, and the layers above are further explained in the Examples section). In the composition abstractions layer more advanced systems with embedded language constructs for composition reside. These systems typically provide a more user-friendly and declarative syntax for composition, e.g., preprocessor directives or aspect syntax. The top-most layer hosts applications that simply use composition tools.

Several specifications are required to develop composition systems in SkAT. The figure below illustrates which ones are provided by SkAT and which ones must be provided to add ISC capabilities to a given fragment language:

Component model specifications

These specifications provide a basic infrastructure and component model based on JastAdd RAGs. The environment grammar defines a common environment that hosts fragment language and component model attributes. The Core specifies attributes for slot and hook identification as well as naming and matching functionality for compositional points, and an implementation of the basic composition operators. Moreover, it adds a basic attribute-based interface for integrating context-sensitive constraints and composition (e.g., based on the static semantics of the fragment language). SkAT/Full adds everything that is required to generate ISC systems as reusable libraries. Full systems support command-mode composition as well as automatic modes using composition strategies. Optionally, by (re)using characteristic attributes in static fragment contracts, wellformedness can be checked continuously during composition. In contrast, SkAT/Minimal supports fragment composition systems that do not or only partially check syntax and semantics. This is beneficial for two reasons. First, creating ISC systems for new fragment languages requires some development effort depending on the language complexity: a language-unaware system can be employed out of the box for any language. Second, supporting inherently complex programming languages, such as C++ or Fortran, using generative technologies is difficult, if not even impossible. However, is possible to support them partially. SkAT/Minimal therefore relies on island grammars and island component models (see also the discussion on scalable ISC above). Hence, parts of a fragment component are recognized as islands and checked during composition while the rest fades in water (as a stream of characters/tokens). Observe that an island grammar formally generates a superset of the related fragment language whereas only islands are reflected as nontrivial subtrees in the generated syntax tree.

Custom fragment-language specifications

To implement an ISC system based on SkAT, according specifications of the fragment language must be provided. In the ideal case, these have already been developed in an extensible way by a language engineer. Otherwise, they additionally have to be created by the composition system developer. To do so, at least a JastAdd AST grammar and a parser emitting JastAdd ASTs need to be provided. SkAT does not make any assumptions about the employed parsing technology. However, if the composition system uses its own embedded syntax, an extensible specification-based approach is favourable. Optionally, if wellformedness is to be checked during composition, JastAdd RAG specifications can be provided that, e.g., implement name analysis and type system of the respective language.

Custom composition system specifications

Finally, for creating composition systems, users of SkAT need to define a language-specific component model. Again, this comprises several attribute-grammar specifications that define occurrences of slots and hooks as well as their specific naming schemes. Moreover, fragment language and component model need to be glued together by providing an integrating AST grammar that basically imports and composes the Core and language grammars under a common composition environment root nonterminal, which is required by the attribute-evaluation engine to compute values of attribute instances. The very basics of component-model specification are discussed in the next section. As this is only a very brief and incomplete documentation, please also refer to the Examples section at the end of this wiki page.

Using SkAT

Specifying component models

Component models in SkAT are, of course, specified using attributes. In the following we will briefly exemplify how component models are specified using JastAdd attribute grammars.

Slots (i.e., variation points) are identified by two synthesized attributes that are declared in Skat/Core including respective default equations that hold for any node in the AST unless something different is defined:

#!java
syn boolean ASTNode.isSlot () = false ; 
syn String ASTNode.slotName () = "" ;

The isSlot attribute defines a predicate that allows SkAT to identify nodes within fragment trees as slots that can be bound (replaced) with another fragment tree. Complementary, slotName defines a name that can be used by users of the resulting composition system to address points during composition. Observe that concrete slot definitions may have an arbitrary complexity as arbitrary Java blocks are allowed in equations.

To understand how slot identification practically works, assume the following Java-like class declaration containing a single method declaration declSlot:

#!java
public class AClass { 
   public void declSlot() {}; 
}

Further assume that we would like the system to identify method declarations (represented by a nonterminal MethodDecl of the fragment-language grammar) whose names end with a suffix Slot and propagate them in the composition environment. This can be achieved by specializing isSlot and slotName for MethodDecls:

#!java
eq MethodDecl.isSlot() = name().endsWith("Slot"); 
eq MethodDecl.slotName() = isSlot()?name().substring(0, name().length() − 4) : "" ;

To compose slots with fragments, adequate fragment types need to be defined. Therefore, SkAT/Core introduces fragment box nonterminals hosted in the CompositionEnvironment (using AST grammars):

#!java
CompositionEnvironment ::= Fragment:Box* /* .. */ ; 
abstract Box ::= <Name:String> <OutName:String> ;

Box is defined as an abstract nonterminal with two string-typed terminal children. Name denotes the name of the fragment, while OutName represents its location in the file system. To adopt fragment types in the composition environment, Box must be specialized accordingly via nonterminal inheritance. For the example above, essentially two kinds of fragment types are required:

#!java

// imported ClassDecl, MethodDecl, Box
ClassBox:Box ::= Fragment:ClassDecl ; 
MethodBox:Box ::= Fragment:MethodDecl ;

Here it is assumed that ClassDecl and MethodDecl are nonterminals of the fragment language. ClassBox and MethodBox adapt them for the composition system via nonterminal inheritance.

Constraining composition through fragment contracts

Additionally, SkAT-based composition systems can check wellformedness during composition using pre- and post- condition attributes that are associated with compositional points and get a fragment tree’s root as an argument:

#!java

syn Object ASTNode.checkContractPre(Object fragment) = true ;
syn Object ASTNode.checkContractPost(Object fragment) = true ;

#!java

syn Object MethodDecl.checkContractPre(Object f){ 
   Set provided = memberDecls() ;
   if(provided.containsAll(f.danglingVars())
      return true ;
   else
   return "Variables missing."
}

Using SkAT composition systems

After generating the composition system using JastAdd, it can be compiled and loaded as a library by any Java application. Applied to our example, this would look as follows:

#!java

CompositionSystem cs = new CompositionSystem( ... ) ; 
cs.addBind("AClass.jbx#decl","int exec(){...}") ; 
cs.triggerComposition() ;

First, the composition system is initialized with a fragment directory and loads AClass.jbx as a ClassBox. Next, a bind operation is declared to replace the decl slot of AClass with an exec method. Third, the composition is executed. The result of the composition then looks as follows:

#!java

public class AClass { 
  int exec(){...}; 
}

Examples

If you are further interested in examples on how SkAT can be used, please have a look into our repository.

As usual: composing Java code

In the spirit of the original COMPOST system (unfortunately its website is down), a complete classic composition system for Java 6 (SkAT4J) can be found in the project org.skat.binding.java. Its specifications folder contains all files of the component model as well as a parser and pretty printer. To compile this system, simply cd into that folder and type ant, which will generate a skat4j.jar file that you can plug into your classpath. Of course you can also use Eclipse for that. To do so, just import the projects in the skat/impl directory into your workspace (without copying!) and right-click on the build.xml to run the build. Applications of SkAT4J can be found in the test-case project org.skat.binding.java.test. Test cases and composition programs used therein are located in the src/org/jastaddj/fragments/test/ folder; the corresponding fragments can be found in test-src/.

Some application test cases are worth having a closer look at:

1. Business application framework: a code generator for a model-based DSL

   • Prerequisits: requires EMFText to generate a parser from the DSL metamodel and grammar. Please install via the Eclipse market place or EMFText update site.
   • Composition program: TestBAFCodeGen.java
   • Input components: business-application-framework/in/
   • Output: business-application-framework/expected/

2. Component-based Software Engineering: course example

   • Composition program: TestCBSE.java
   • Input components: cbse/in/
   • Output: cbse/expected/

3. Parametrizable logging aspect: the most trivial application of aspect-oriented programming

   • Composition program: TestLoggingAspect.java
   • Input components: logging-aspect/in/
   • Output: logging-aspect/expected/

4. Mixin-based inheritance: the famous mixin operator for multiple inheritance in Java (also used in other examples)

   • Composition program: TestMixin.java
   • Input components: logging-aspect/in/
   • Output: logging-aspect/expected/

5. Map/Reduce/MapReduce skeleton library: composing parallel applications using well-formed ISC

   • Composition program: TestSkeletons.java
   • Input components: count-and-sort/in/
   • Output: count-and-sort/expected/ (4 variants) and count-and-sort/expected2/ (2 variants)

Minimal and partially safe composition systems

1. SkAT Minimal: an extremely simple composition system only with support for slot markup, independent of a specific fragment language. The capabilities of this system can be summarized as follows: given a string " ... #Slot1# ... #Slot2# ... #Slotn#..." it supports the replacement of the Sloti with arbitrary strings.

   • Component model: specifications folder of org.skat.isc.minimal/

2. Variability Template Language (VTpL): a verbose extension of SkAT Minimal with markup for variants and prototypes in the code. [[PROTOTYPE::name]] tags are provided to mark-up code that acts as an "embedded" fragment component that can be instaiated multiple times by the composition system. [[VARIANTS::name]] tags group lists of code variants that can be chosen alternatively by a composition program. Note that VTpL does not contain any control-flow statements. As it is normal in classic ISC, control-flow is kept solely in separate composition programs, which makes it possible to generate code in a non-linear fashion.

   • Component model: specifications folder of org.skat.binding.vtpl/
   • Example composition programs: TemplateEngineTest.java and ThesisExamples.java
   • Example fragment components: testdata folder of org.skat.binding.vtpl.test/

3. Univeral (Extensible) Preprocessor (UPP): an extension of SkAT Minimal that resembles the well-known c preprocessor (CPP) and provides an extensible component model towards partially safe code generation based on ISC. In contrast to the applications discussed so far, UPP has an embedded composition syntax and its own control flow and thus is a command-line tool rather than a library.

   • Component model: specifications folder of org.skat.binding.upp/
   • Example composition programs: method testProcessing in ProcessingTest.java.
   • Example fragment components: org.skat.binding.upp.test/testdata/processing/in/

4. Fortran Preprocessor (FPP): an extension of SkAT Minimal that recognizes do concurrent constructs in Fortran programs and transforms them to a regular do loop with OpenMP pragmas using invasive software composition.

   • Component model: specifications folder of org.skat.binding.fpp/

Further Reading

If you are eager to learn more about the method of invasive software composition, the approach of SkAT and its relation to previous approaches to ISC, you are kindly referred to my thesis and presentation slides, which are both published via an open-access platform and therefore free of charge.