Serialization

Serialization is the process of transforming an EMF model into its textual representation. Thereby, serialization complements parsing and lexing.

In Xtext, the process of serialization is split into the following steps:

  1. Validating the semantic model. This is optional, enabled by default, done by the concrete syntax validator and can be turned off in the save options.

  2. Matching the model elements with the grammar rules and creating a stream of tokens. This is done by the parse tree constructor.

  3. Associating comments with semantic objects. This is done by the comment associator.

  4. Associating existing nodes from the node model with tokens from the token stream.

  5. Merging existing whitespace and line-wraps into to token stream.

  6. Adding further needed whitespace or replacing all whitespace using a formatter.

Serialization is invoked when calling XtextResource.save(...). Furthermore, Serializer provides resource-independent support for serialization. Serialization is not called when a textual editors contents is saved to disk. Another situation that triggers serialization is applying Quick Fixes with semantic modifications.

The Contract

The contract of serialization says that a model that is serialized to its textual representation and then loaded (parsed) again should yield a loaded model that equals the original model. Please be aware that this does not imply, that loading a textual representation and serializing it back produces identical textual representations. For example, this is the case when a default value is used in a textual representation and the assignment is optional. Another example is:

MyRule:
  (xval+=ID | yval+=INT)*;
  

MyRule in this example reads ID- and INT-elements which may occur in an arbitrary order in the textual representation. However, when serializing the model all ID-elements will be written first and then all INT-elements. If the order is important it can be preserved by storing all elements in the same list – which may require wrapping the ID- and INT-elements into objects.

Roles of the Semantic Model and the Node Model During Serialization

A serialized document represents the state of the semantic model. However, if there is a node model available (i.e. the semantic model has been created by the parser), the serializer

Parse Tree Constructor

The parse tree constructor usually does not need to be customized since it is automatically derived from the Xtext Grammar. However, it can be helpful to look into it to understand its error messages and its runtime performance.

For serialization to succeed, the parse tree constructor must be able to consume every non-transient element of the to-be-serialized EMF model. To consume means, in this context, to write the element to the textual representation of the model. This can turn out to be a not-so-easy-to-fulfill requirement, since a grammar usually introduces implicit constraints to the EMF model as explained for the concrete syntax validator.

If a model can not be serialized, an XtextSerializationException is thrown. Possible reasons are listed below:

To understand error messages and performance issues of the parse tree constructor it is important to know that it implements a backtracking algorithm. This basically means that the grammar is used to specify the structure of a tree in which one path (from the root node to a leaf node) is a valid serialization of a specific model. The parse tree constructor’s task is to find this path – with the condition, that all model elements are consumed while walking this path. The parse tree constructor’s strategy is to take the most promising branch first (the one that would consume the most model elements). If the branch leads to a dead end (for example, if a model element needs to be consumed that is not present in the model), the parse tree constructor goes back the path until a different branch can be taken. This behavior has two consequences:

Options

SaveOptions can be passed to XtextResource.save(options) and to Serializer.serialize(..). Available options are:

Preserving Comments from the Node Model

The ICommentAssociater associates comments with semantic objects. This is important in case an element in the semantic model is moved to a different position and the model is serialized, one expects the comments to be moved to the new position in the document as well.

Which comment belongs to which semantic object is surely a very subjective issue. The default implementation ( DefaultCommentAssociater) behaves as follows, but can be customized:

Transient Values

Transient values are values or model elements which are not persisted (written to the textual representation in the serialization phase). If a model contains model elements which can not be serialized with the current grammar, it is critical to mark them transient using the ITransientValueService, or serialization will fail. The default implementation marks all model elements transient, which are eStructuralFeature.isTransient() or not eObject.eIsSet(eStructuralFeature). By default, EMF returns false for eIsSet(...) if the value equals the default value.

Unassigned Text

If there are calls of data type rules or terminal rules that do not reside in an assignment, the serializer by default doesn’t know which value to use for serialization.

. Example:

PluralRule:
  'contents:' count=INT Plural;
  
terminal Plural: 
  'item' | 'items';
  

Valid models for this example are contents 1 item or contents 5 items. However, it is not stored in the semantic model whether the keyword item or items has been parsed. This is due to the fact that the rule call Plural is unassigned. However, the parse tree constructor needs to decide which value to write during serialization. This decision can be be made by customizing the IValueSerializer.serializeUnassignedValue(EObject, RuleCall, AbstractNode).

Cross-Reference Serializer

The cross-reference serializer specifies which values are to be written to the textual representation for cross-references. This behavior can be customized by implementing ICrossReferenceSerializer. The default implementation delegates to various other services such as the IScopeProvider or the LinkingHelper each of which may be the better place for customization.

Merge Whitespaces

After the parse tree constructor has done its job to create a stream of tokens which are to be written to the textual representation, and the comment associator has done its work, existing whitespace form the node model is merged into the stream.

The strategy is as follows: If two tokens follow each other in the stream and the corresponding nodes in the node model follow each other as well, then the whitespace information in between is kept. In all other cases it is up to the formatter to calculate new whitespace information.

Token Stream

The parse tree constructor and the formatter use an ITokenStream for their output, and the latter for its input as well. This makes them chainable. Token streams can be converted to a String using the TokenStringBuffer and to a Writer using the WriterTokenStream.

public interface ITokenStream {
  void flush() throws IOException;
  void writeHidden(EObject grammarElement, String value) throws IOException;
  void writeSemantic(EObject grammarElement, String value) throws IOException;
}