Package eu.bandm.tools.lexic

Class TokenRuleSet<T>

java.lang.Object
eu.bandm.tools.lexic.TokenRuleSet<T>
Type Parameters:
T - the type of token types
All Implemented Interfaces:
Serializable
public class TokenRuleSet<T> extends Object implements Serializable
A set of token rules together with a precendence relation between token types.

An object of this class completely defines the lexical structure of a formal language.

The given token rules are combined disjunctively, analogously to TokenFragment.orElse(TokenFragment).

Precedence

A precendence relation can be given in order to choose between different token types when more than one matches. The precedence relation is specified by a precedence oracle, a BiPredicate on token types. Token type t has immediate precedence over token type u if the oracle, applied to (t, u), returns true. The relation of precedence is the reflexive and transitive closure of immediate precedence. Among a collection of token types, a token type is foremost, if no other token type in the collection has precedence over it. Note that every non-empty collection has at least one foremost element.

Whenever several token types match a given input, only the foremost ones are retained. If only one token type remains, then the situation has been successfully disambiguated. Otherwise, an error, possibly either in the input or in the set of token rules, may be signaled.

A typical usage example is the relation where token types associated with a fixed input string have immediate precedence over token types associated with variable input strings. This subsumes the precedence of keywords over identifiers.

Serialization

If an object of this class is to be serialized, make sure that the precedence oracle and all token types are completely serializable. This cannot be enforced statically in a non-obtrusive way.

See Also:

  • Constructor Details

    • TokenRuleSet

      @SafeVarargs public TokenRuleSet(TokenRule<T>... rules)
      Creates a new instance with the given rules and the empty immediate precedence relation.
      Parameters:
      rules - the token rules

    • TokenRuleSet

      @SafeVarargs public TokenRuleSet(BiPredicate<? super T,? super T> precedenceOracle, TokenRule<T>... rules)
      Creates a new instance with the given rules and immediate precedence relation.
      Parameters:
      precedenceOracle - a predicate that encodes the immediate precedence relation
      rules - the token rules

    • TokenRuleSet

      public TokenRuleSet(Collection<? extends TokenRule<T>> rules)
      Creates a new instance with the given rules and the empty immediate precedence relation.
      Parameters:
      rules - the token rules

    • TokenRuleSet

      public TokenRuleSet(BiPredicate<? super T,? super T> precedenceOracle, Collection<? extends TokenRule<T>> rules)
      Creates a new instance with the given rules and immediate precedence relation.
      Parameters:
      precedenceOracle - a predicate that encodes the immediate precedence relation
      rules - the token rules

  • Method Details

    • getRules

      public Set<TokenRule<T>> getRules()
      Returns the set of token rules.
      Returns:
      an unmodifiable set containing the token rules

    • filterByPrecedence

      public List<T> filterByPrecedence(Collection<T> types)
      Returns the preferred token types from a given collection according to the precedence relation.
      Parameters:
      types - a collection of token types
      Returns:
      a list containing only the preferred among the given token types