Package eu.bandm.tools.lexic

Interface LookaheadTokenSource<D,T,L>

Type Parameters:
D - the type of source document identifiers
T - the type of token types
L - the type of lookahead data
All Superinterfaces:
Supplier<Token<D,T>>
All Known Subinterfaces:
TokenSource<D,T>
All Known Implementing Classes:
LookaheadTokenFilter, LookaheadTokenMultiplexer, LookaheadTokenProcessor, Screener, TokenFilter, TokenProcessor
public interface LookaheadTokenSource<D,T,L> extends Supplier<Token<D,T>>
A supplier of tokens with internal lookahead buffer.

Token sources must not return null or throw an exception to indicate that there are no more tokens available. Instead, it must supply an infinite number of special tokens that indicate the end of the input stream. Instead of creating multiple such tokens, a token source may return multiple references to the same token object. How the client recognizes end-of-stream tokens is application-dependent; it is recommended that a distinguished token type be defined for the purpose.

Lookahead data can be transferred from the internal buffer of one instance of this interface to another. This is useful for transfer of control if both share the same input source.

  • Method Details

    • relinquishLookahead

      L relinquishLookahead()
      Clear the internal lookahead buffer and return its contents.
      Returns:
      the former contents of the internal lookahead buffer

    • takeOverLookahead

      void takeOverLookahead(L lookaheadData)
      Fill the internal lookahead buffer with data.

      The buffer must be empty before this method is invoked. After return, the given data will be consumed from the buffer before the usual input is considered.

      Parameters:
      lookaheadData - the lookahead data to fill in
      Throws:
      IllegalStateException - if the internal lookahead buffer is not empty

    • transferLookahead

      static <A> void transferLookahead(LookaheadTokenSource<?,?,? extends A> first, LookaheadTokenSource<?,?,? super A> second)

    • forgetLookahead

      default TokenSource<D,T> forgetLookahead()
      Returns a token source that forwards tokens and does not use lookahead.
      Returns:
      a token source that forwards tokens from this token source

    • removeTypes

      default LookaheadTokenSource<D,T,L> removeTypes(T... types)
      Returns a secondary token source that forwards all tokens from this token source that have none of the given types.
      Parameters:
      types - the types of tokens to remove
      Returns:
      a token source that forwards all tokens from this token source that have none of the given types

    • removeTypes

      default LookaheadTokenSource<D,T,L> removeTypes(Set<T> types)
      Returns a secondary token source that forwards all tokens from this token source that have none of the given types.
      Parameters:
      types - the types of tokens to remove
      Returns:
      a token source that forwards all tokens from this token source that have none of the given types