Package eu.bandm.tools.lexic

Interface TokenSource<D,T>

Type Parameters:
D - the type of source document identifiers
T - the type of token types
All Superinterfaces:
LookaheadTokenSource<D,T,Void>, Supplier<Token<D,T>>
All Known Implementing Classes:
TokenFilter, TokenProcessor
Functional Interface:
This is a functional interface and can therefore be used as the assignment target for a lambda expression or method reference.
@CyclicDependency @FunctionalInterface public interface TokenSource<D,T> extends LookaheadTokenSource<D,T,Void>
A specialized supplier of tokens.

Token sources must not return null or throw an exception to indicate that there are no more tokens available. Instead, it must supply an infinite number of special tokens that indicate the end of the input stream. Instead of creating multiple such tokens, a token source may return multiple references to the same token object. How the client recognizes end-of-stream tokens is application-dependent; it is recommended that a distinguished token type be defined for the purpose.

See Also:

  • Method Details

    • relinquishLookahead

      default Void relinquishLookahead()
      Description copied from interface: LookaheadTokenSource
      Clear the internal lookahead buffer and return its contents.
      Specified by:
      relinquishLookahead in interface LookaheadTokenSource<D,T,Void>
      Returns:
      the former contents of the internal lookahead buffer

    • takeOverLookahead

      default void takeOverLookahead(Void lookaheadData)
      Description copied from interface: LookaheadTokenSource
      Fill the internal lookahead buffer with data.

      The buffer must be empty before this method is invoked. After return, the given data will be consumed from the buffer before the usual input is considered.

      Specified by:
      takeOverLookahead in interface LookaheadTokenSource<D,T,Void>
      Parameters:
      lookaheadData - the lookahead data to fill in

    • with

      default TokenSource<D,T> with(Function<? super TokenSource<D,T>,? extends TokenProcessor<D,T>> processor)
      Creates a new token processor by applying a given factory function to this source.
      Parameters:
      processor - the factory function for creating a processor
      Returns:
      a new token processor by applying the given factory function to this source

    • removeTypes

      default TokenSource<D,T> removeTypes(T... types)
      Returns a secondary token source that forwards all tokens from this token source that have none of the given types.
      Specified by:
      removeTypes in interface LookaheadTokenSource<D,T,Void>
      Parameters:
      types - the types of tokens to remove
      Returns:
      a token source that forwards all tokens from this token source that have none of the given types

    • removeTypes

      default TokenSource<D,T> removeTypes(Set<T> types)
      Returns a secondary token source that forwards all tokens from this token source that have none of the given types.
      Specified by:
      removeTypes in interface LookaheadTokenSource<D,T,Void>
      Parameters:
      types - the types of tokens to remove
      Returns:
      a token source that forwards all tokens from this token source that have none of the given types