Package eu.bandm.tools.lexic

Interface LookaheadTokenSource<D,T,L>

Type Parameters:
D - the type of source document identifiers
T - the type of token types
L - the type of lookahead data
All Superinterfaces:
AutoCloseable, Closeable, Supplier<Token<D,T>>
All Known Subinterfaces:
TokenSource<D,T>
All Known Implementing Classes:
LookaheadTokenFilter, LookaheadTokenMultiplexer, LookaheadTokenProcessor, LookaheadTokenSourceProxy, Screener, TokenFilter, TokenProcessor
public interface LookaheadTokenSource<D,T,L> extends Supplier<Token<D,T>>, Closeable
A supplier of tokens with internal lookahead buffer.

Token sources must not return null or throw an exception to indicate that there are no more tokens available. Instead, it must supply an infinite number of special tokens that indicate the end of the input stream. Instead of creating multiple such tokens, a token source may return multiple references to the same token object. How the client recognizes end-of-stream tokens is application-dependent; it is recommended that a distinguished token type be defined for the purpose.

Lookahead data can be transferred from the internal buffer of one instance of this interface to another. This is useful for transfer of control if both share the same input source.

This interface extends Closeable, but the default implementation of close() does nothing. Implementing classes are assumed by default not to expect explicit closing. An implementing class for which closing is beneficial or even required should be documented explicitly as such. Note that this interfaces restricts the contract of close() to not throw IOException.

  • Method Summary

    Modifier and Type
    Method
    Description
    default void
    close()
    Closes this source and releases any resources associated with it.
    default TokenSource<D,T>
    forgetLookahead()
    Returns a token source that forwards tokens and does not use lookahead.
    L
    relinquishLookahead()
    Clear the internal lookahead buffer and return its contents.
    default LookaheadTokenSource<D,T,L>
    removeTypes(Set<T> types)
    Returns a secondary token source that forwards all tokens from this token source that have none of the given types.
    default LookaheadTokenSource<D,T,L>
    removeTypes(T... types)
    Returns a secondary token source that forwards all tokens from this token source that have none of the given types.
    void
    takeOverLookahead(L lookaheadData)
    Fill the internal lookahead buffer with data.
    static <L> void
    transferLookahead(LookaheadTokenSource<?,?,? extends L> first, LookaheadTokenSource<?,?,? super L> second)
    Transfers lookahead buffer data between two compatible token sources.

    Methods inherited from interface java.util.function.Supplier

    get

  • Method Details

    • relinquishLookahead

      L relinquishLookahead()
      Clear the internal lookahead buffer and return its contents.
      Returns:
      the former contents of the internal lookahead buffer

    • takeOverLookahead

      void takeOverLookahead(L lookaheadData)
      Fill the internal lookahead buffer with data.

      The buffer must be empty before this method is invoked. After return, the given data will be consumed from the buffer before the usual input is considered.

      Parameters:
      lookaheadData - the lookahead data to fill in
      Throws:
      IllegalStateException - if the internal lookahead buffer is not empty

    • transferLookahead

      static <L> void transferLookahead(LookaheadTokenSource<?,?,? extends L> first, LookaheadTokenSource<?,?,? super L> second)
      Transfers lookahead buffer data between two compatible token sources.

      This method calls relinquishLookahead() on the first source, followed by takeOverLookahead(L) on the second.

      Two sources are compatible if the lookahead data type of the first is assignable to the lookahead data type of the second. Source document identifier and token types are irrelevant.

      Type Parameters:
      L - the type of lookahead data
      Parameters:
      first - the first token source
      second - token source
      See Also:

    • forgetLookahead

      default TokenSource<D,T> forgetLookahead()
      Returns a token source that forwards tokens and does not use lookahead.
      Returns:
      a token source that forwards tokens from this token source

    • removeTypes

      default LookaheadTokenSource<D,T,L> removeTypes(T... types)
      Returns a secondary token source that forwards all tokens from this token source that have none of the given types.
      Parameters:
      types - the types of tokens to remove
      Returns:
      a token source that forwards all tokens from this token source that have none of the given types

    • removeTypes

      default LookaheadTokenSource<D,T,L> removeTypes(Set<T> types)
      Returns a secondary token source that forwards all tokens from this token source that have none of the given types.
      Parameters:
      types - the types of tokens to remove
      Returns:
      a token source that forwards all tokens from this token source that have none of the given types

    • close

      default void close()
      Closes this source and releases any resources associated with it.

      The default implementation does nothing.

      Specified by:
      close in interface AutoCloseable
      Specified by:
      close in interface Closeable