Package eu.bandm.tools.lexic

Interface TokenSource<D,T>

Type Parameters:
D - the type of source document identifiers
T - the type of token types
All Superinterfaces:
AutoCloseable, Closeable, LookaheadTokenSource<D,T,Void>, Supplier<Token<D,T>>
All Known Implementing Classes:
TokenFilter, TokenProcessor
Functional Interface:
This is a functional interface and can therefore be used as the assignment target for a lambda expression or method reference.
@CyclicDependency @FunctionalInterface public interface TokenSource<D,T> extends LookaheadTokenSource<D,T,Void>
A specialized supplier of tokens.

Token sources must not return null or throw an exception to indicate that there are no more tokens available. Instead, it must supply an infinite number of special tokens that indicate the end of the input stream. Instead of creating multiple such tokens, a token source may return multiple references to the same token object. How the client recognizes end-of-stream tokens is application-dependent; it is recommended that a distinguished token type be defined for the purpose.

See Also:

  • Method Summary

    Modifier and Type
    Method
    Description
    default TokenSource<D,T>
    parallel(int capacity)
    Returns a secondary token source that forwards tokens from this token source asynchronously in a new worker thread.
    default Void
    relinquishLookahead()
    Clear the internal lookahead buffer and return its contents.
    default TokenSource<D,T>
    removeTypes(Set<T> types)
    Returns a secondary token source that forwards all tokens from this token source that have none of the given types.
    default TokenSource<D,T>
    removeTypes(T... types)
    Returns a secondary token source that forwards all tokens from this token source that have none of the given types.
    default void
    takeOverLookahead(Void lookaheadData)
    Fill the internal lookahead buffer with data.
    default TokenSource<D,T>
    with(Function<? super TokenSource<D,T>,? extends TokenProcessor<D,T>> processor)
    Creates a new token processor by applying a given factory function to this source.

    Methods inherited from interface eu.bandm.tools.lexic.LookaheadTokenSource

    close, forgetLookahead

    Methods inherited from interface java.util.function.Supplier

    get

  • Method Details

    • relinquishLookahead

      default Void relinquishLookahead()
      Description copied from interface: LookaheadTokenSource
      Clear the internal lookahead buffer and return its contents.
      Specified by:
      relinquishLookahead in interface LookaheadTokenSource<D,T,Void>
      Returns:
      the former contents of the internal lookahead buffer

    • takeOverLookahead

      default void takeOverLookahead(Void lookaheadData)
      Description copied from interface: LookaheadTokenSource
      Fill the internal lookahead buffer with data.

      The buffer must be empty before this method is invoked. After return, the given data will be consumed from the buffer before the usual input is considered.

      Specified by:
      takeOverLookahead in interface LookaheadTokenSource<D,T,Void>
      Parameters:
      lookaheadData - the lookahead data to fill in

    • with

      default TokenSource<D,T> with(Function<? super TokenSource<D,T>,? extends TokenProcessor<D,T>> processor)
      Creates a new token processor by applying a given factory function to this source.
      Parameters:
      processor - the factory function for creating a processor
      Returns:
      a new token processor by applying the given factory function to this source

    • removeTypes

      default TokenSource<D,T> removeTypes(T... types)
      Returns a secondary token source that forwards all tokens from this token source that have none of the given types.
      Specified by:
      removeTypes in interface LookaheadTokenSource<D,T,Void>
      Parameters:
      types - the types of tokens to remove
      Returns:
      a token source that forwards all tokens from this token source that have none of the given types

    • removeTypes

      default TokenSource<D,T> removeTypes(Set<T> types)
      Returns a secondary token source that forwards all tokens from this token source that have none of the given types.
      Specified by:
      removeTypes in interface LookaheadTokenSource<D,T,Void>
      Parameters:
      types - the types of tokens to remove
      Returns:
      a token source that forwards all tokens from this token source that have none of the given types

    • parallel

      default TokenSource<D,T> parallel(int capacity)
      Returns a secondary token source that forwards tokens from this token source asynchronously in a new worker thread.

      Tokens may be produced in advance by the worker, up to a given capacity.

      This construct is useful for high-throughput scenarios, because worker and consumer can be executed on different CPU cores. Because of asynchronous computation, it does not support dynamic changes to the token pipeline, such as LookaheadTokenSource.transferLookahead(eu.bandm.tools.lexic.LookaheadTokenSource<?, ?, ? extends L>, eu.bandm.tools.lexic.LookaheadTokenSource<?, ?, ? super L>).

      The worker thread is started right away as a daemon thread. When the consumer ceases to consume tokens, the worker will fill the queue up to its capacity, and then block indefinitely. Calling the LookaheadTokenSource.close() method of the parallel token source will interrupt the worker thread, and thus cause it to terminate as soon as possible. This is recommended for releasing thread-related resources.

      Parameters:
      capacity - the maximum number of tokens that may be produced in advance
      Returns:
      a token source that forwards tokens from this token source asynchronously in a new worker thread
      Throws:
      IllegalArgumentException - if capacity is not positive
      See Also: