Five simple steps to your JVM language

In this tutorial we will basically implement the domain model language again, but this time, we will make use of the special JVM support shipped with Xtext 2.1. This kind of language really is a sweet spot for DSLs, so feel free to use this as a blueprint and add your project specific features later on.

The revised domain model language supports expressions and cross links to Java types. It is directly translated to Java source code. The syntax should look very familiar. Here is an example :

import java.util.List

package my.model {

    entity Person {
        name: String
        firstName: String
        friends: List<Person>
        address : Address
        op getFullName() : String {
            return firstName + " " + name;
        }
        
        op getFriendsSortedByFullName() : List<Person> {
            return friends.sortBy( f | f.fullName);
        }
    }
    
    entity Address {
        street: String
        zip: String
        city: String
    }
}

As you can see, it supports all kinds of advanced features such as Java generics and full expressions even including closures. Don't panic you'll not have to implement these concepts on your own but will reuse a lot of helpful infrastructure to build the language.

We will now walk through the five! little steps needed to get this language fully working including its compiler.

After you have installed Xtext on your machine, start Eclipse and set up a fresh workspace.

Step One: Create A New Xtext Project

In order to get started we first need to create some Eclipse projects. Use the Eclipse wizard to do so:

File -> New -> Project... -> Xtext -> Xtext project

Choose a meaningful project name, language name and file extension, e.g.

Main project name: org.example.domainmodel
Language name: org.example.domainmodel.Domainmodel
DSL-File extension: dmodel

Click on Finish to create the projects.

After you've successfully finished the wizard, you'll find three new projects in your workspace.

org.example.domainmodel Contains the grammar definition and all runtime components (parser, lexer, linker, validation, etc.)
org.example.domainmodel.tests Unit tests go here.
org.example.domainmodel.ui The Eclipse editor and all the other workbench related functionality.

Step Two: Write the Grammar

The wizard will automatically open the grammar file Domainmodel.xtext in the editor. As you can see it already contains a simple Hello World grammar:

grammar org.example.domainmodel.Domainmodel with
                                      org.eclipse.xtext.common.Terminals

generate domainmodel "http://www.example.org/domainmodel/Domainmodel"

Model:
    greetings+=Greeting*;
    
Greeting:
    'Hello' name=ID '!';

Please replace that grammar definition with the one for our language:

grammar org.example.domainmodel.DomainModel with
                                      org.eclipse.xtext.xbase.Xbase

generate domainmodel "http://www.example.org/domainmodel/Domainmodel"

DomainModel:
    elements+=AbstractElement*;

AbstractElement:
    PackageDeclaration | Entity | Import;

PackageDeclaration:
    'package' name=QualifiedName '{'
        elements+=AbstractElement*
    '}';

Import:
    'import' importedNamespace=QualifiedNameWithWildCard;

QualifiedNameWithWildCard :
    QualifiedName  ('.' '*')?;

Entity:
    'entity' name=ValidID 
        ('extends' superType=JvmTypeReference)? '{'
        features+=Feature*
    '}';

Feature:
    Property | Operation;

Property:
    name=ValidID ':' type=JvmTypeReference;

Operation:
    'op' name=ValidID 
        '('(params+=FullJvmFormalParameter 
            (',' params+=FullJvmFormalParameter)*)?')'
        ':' type=JvmTypeReference 
        body=XBlockExpression;

Let's have a look at what the different grammar constructs mean:

  1. grammar org.example.domainmodel.DomainModel with
                                org.eclipse.xtext.xbase.Xbase

    The first thing to note is that instead of inheriting from the usual org.eclipse.xtext.common.Terminals grammar, we make use of org.eclipse.xtext.xbase.Xbase. Xbase allows us to easily reuse and embed modern, statically typed expressions as well as Java type signatures in our language. In case you also want to use Java annotations, you can extend org.eclipse.xtext.xbase.XbaseWithAnnotations instead.
  2. DomainModel:
        elements+=AbstractElement*;

    The first rule in a grammar is always used as the entry or start rule. It says that a DomainModel contains an arbitrary number (*) of AbstractElements which will be added (+=) to a feature called elements.
  3. AbstractElement:
        PackageDeclaration | Entity | Import;

    The rule AbstractElement delegates to either the rule PackageDeclaration, the rule Entity or the rule Import.
  4. PackageDeclaration:
        'package' name=QualifiedName '{'
            elements+=AbstractElement*
        '}';

    A PackageDeclaration is used to declare a name space which can again contain any number of AbstractElements. Xtext has built-in support for qualified names and scoping based on the hierarchy of the produced model. The default implementation will add the package names as the prefix to contained entities and nested packages. The qualified name of an Entity 'Baz' which is contained in a PackageDeclaration 'foo.bar' will be 'foo.bar.Baz'. In case you don't like the default behavior you'll need to use a different implementation of IQualifiedNameProvider (src).
  5. Import:
        'import' importedNamespace=QualifiedNameWithWildCard;

    QualifiedNameWithWildCard :
        QualifiedName  ('.' '*')?;

    The rule Import makes use of the namespace support, too. It basically allows you to get full-blown import functionality as you are used to from Java, just by having these two rules in place.
  6. Entity:
        'entity' name=ValidID 
            ('extends' superType=JvmTypeReference)? '{'
            features+=Feature*
        '}';

    The rule Entity starts with the definition of a keyword followed by a name. The extends clause which is parenthesized and optional (note the trailing ?) makes use of the rule JvmTypeReference which is defined in a super grammar. JvmTypeReference defines the syntax for full Java-like type names. That is everything from simple names, over fully qualified names to fully-fledged generics, including wildcards, lower bounds and upper bounds. Finally between curly braces there can be any number of Features, which leads us to the next rule.
  7. Feature:
        Property | Operation;

    The rule Feature delegates to either a Property or an Operation.
  8. Property:
        name=ValidID ':' type=JvmTypeReference;

    A Property has a name and makes again use of the inherited rule JvmTypeReference.
  9. Operation:
        'op' name=ValidID 
        '('(params+=FullJvmFormalParameter 
            (',' params+=FullJvmFormalParameter)*)?')'
        ':' type=JvmTypeReference 
        body=XBlockExpression;

    Operation's also have a signature as expected. Note that also for formal parameters we can reuse a rule from the super grammar. The Operation's body, that is the actual implementation is defined by the rule XBlockExpression which is one of the most often used entry rules from Xbase. A block consist of any number of expressions between curly braces. Example:

    {
      return "Hello World" + "!"
    }

Step Three: Generate Language Artifacts

Now that we have the grammar in place and defined we need to execute the code generator that will derive the various language components. To do so right click in the grammar editor. From the opened context menu, choose

Run As -> Generate Xtext Artifacts.

This will trigger the Xtext language generator. It generates the parser and serializer and some additional infrastructure code. You will see its logging messages in the Console View.

Step Four: Define the Mapping to JVM Concepts

The syntax alone is not enough to make the language work. We need to map the domain specific concepts to some other language in order to tell Xtext how it is executed. Usually you define a code generator or an interpreter for that matter, but languages using Xbase can omit this step and make use of the IJvmModelInferrer (src).

The idea is that you translate your language concepts to any number of Java types (JvmDeclaredType (src) ). Such a type can be a Java class, Java interface, Java annotation type or a Java enum and may contain any valid members. In the end you as a language developer are responsible to create a correct model according to the Java language.

By mapping your language concepts to Java elements, you implicitly tell Xtext in what kind of scopes the various expressions live and what return types are expected from them. Xtext 2.1 also comes with a code generator which can translate that Java model into readable Java code, including the expressions.

If you have already triggered the 'Generate Xtext Artifacts' action, you should find a stub called org/example/domainmodel/jvmmodel/DomainModelJvmModelInferrer.xtend in the src folder. Please replace its contents with the following :

package org.example.domainmodel.jvmmodel

import com.google.inject.Inject
import org.eclipse.xtext.common.types.JvmDeclaredType
import org.eclipse.xtext.naming.IQualifiedNameProvider
import org.eclipse.xtext.util.IAcceptor
import org.eclipse.xtext.xbase.jvmmodel.AbstractModelInferrer
import org.eclipse.xtext.xbase.jvmmodel.JvmTypesBuilder
import org.example.domainmodel.domainmodel.Entity
import org.example.domainmodel.domainmodel.Operation
import org.example.domainmodel.domainmodel.Property

class DomainModelJvmModelInferrer extends AbstractModelInferrer {

  /**
   * a builder API to programmatically create Jvm elements 
   * in readable way.
   */

  @Inject extension JvmTypesBuilder
  
  @Inject extension IQualifiedNameProvider
  
  def dispatch void infer(Entity element, 
                IAcceptor<JvmDeclaredType> acceptor, 
                boolean isPrelinkingPhase) {
    
    acceptor.accept(element.toClass(element.fullyQualifiedName) [
      documentation = element.documentation
      for (feature : element.features) {
        switch feature {
          Property : {
            members += feature.toField(feature.name, feature.type)
            members += feature.toSetter(feature.name, feature.type)
            members += feature.toGetter(feature.name, feature.type)
          }
          Operation : {
            members += feature.toMethod(feature.name, feature.type) [
              for (p : feature.params) {
                parameters += p.toParameter(p.name, p.parameterType)
              }
              documentation = feature.documentation
              body = feature.body
            ]
          }
        }
      }
    ])
  }
}

Let's go through the code to get an idea of what's going on. (Please also refer to the JavaDoc of the involved API, especially the JvmTypesBuilder (src).)

  1. def dispatch void infer(Entity element, 
              IAcceptor<JvmDeclaredType> acceptor, 
              boolean isPrelinkingPhase) { // (1)

    Using the dispatch keyword makes sure that the method is called for instances of type Entity only. Have a look at polymorphic dispatch to understand Xtend's dispatch functions. Extending AbstractModelInferrer (src) makes sure we don't have to walk the syntax model on our own.
  2. acceptor.accept(element.toClass(element.fullyQualifiedName) [ 
    ...

    Every JvmDeclaredType (src) you create in the model inference needs to be passed to the acceptor in order to get recognized. The extension method toClass comes from JvmTypesBuilder (src). That class provides a lot of convenient extension methods, which help making the code extremely readable and concise. Most of the methods accept initializer blocks as the last argument, in which the currently created model element is bound to the implicit variable it. Therein you can further initialize the created Java element.
  3. documentation = element.documentation Here for instance we assign some JavaDoc to the newly created element. The assignment is translated to an invocation of the method JvmTypesBuilder (src)#setDocumentation(JvmIdentifiableElement (src) element,String documentation) and element.documentation is in fact calling the extension method JvmTypesBuilder (src)#getDocumentation(EObject element) Xtend's extension methods are explained in detail in the sections about extension methods and imports.
  4. for (feature : element.features) {
      switch feature { // (4)
        Property : {
          // ...
        }
        Operation : {
          // ...
        }
      }
    }

    When iterating over a list of heterogeneous types, the switch expression with its type guards comes in handy. If feature is of type Property the first block is executed. If it is an Operation the second block is executed. Note that the variable feature will be implicitly casted to the respective type within the blocks.
  5. Property : {
      members += feature.toField(feature.name, feature.type) // (5)
      members += feature.toSetter(feature.name, feature.type)
      members += feature.toGetter(feature.name, feature.type)
    }

    For each Property we create a field as well as a corresponding getter and setter.
  6. Operation : {
      members += feature.toMethod(feature.name, feature.type) [
        for (p : feature.params) {
          parameters += p.toParameter(p.name, p.parameterType)
        }
        documentation = feature.documentation
        body = feature.body
      ]
    }

    Operations are being mapped to a corresponding Java method. The documentation is translated and the parameters are added within the initializer. The line body = feature.body registers the Operation's expression as the body of the newly created Java method. This defines the scope of the expression. The frameworks deduces the visible fields and parameters as well as the expected return type from that information.

Step Five : Try the Editor!

We are now able to test the IDE integration, by spawning a new Eclipse using our plug-ins. To do so just use the launch shortcut called "Launch Runtime Eclipse", clicking on the green play button in the tool bar.

In the new workbench, create a Java project (File -> New -> Project... -> Java Project and therein a new file with the file extension you chose in the beginning (*.dmodel). This will open the generated entity editor. Try it and discover the rich functionality it provides. You should also have a look at the preferences of your language to find out what can be individually configured to your users needs.

Have fun!