Package eu.bandm.tools.lexic

Interface TokenSource<D,T>

Type Parameters:
D - the type of source document identifiers
T - the type of token types
All Superinterfaces:
Supplier<Token<D,T>>
All Known Implementing Classes:
TokenFilter, TokenProcessor
@CyclicDependency public interface TokenSource<D,T> extends Supplier<Token<D,T>>
A specialized supplier of tokens.

Token sources must not return null or throw an exception to indicate that there are no more tokens available. Instead, it must supply an infinite number of special tokens that indicate the end of the input stream. Instead of creating multiple such tokens, a token source may return multiple references to the same token object. How the client recognizes end-of-stream tokens is application-dependent; it is recommended that a distinguished token type be defined for the purpose.

See Also:

  • Method Summary

    Modifier and Type
    Method
    Description
    default TokenSource<D,T>
    removeTypes(Set<T> types)
    Returns a secondary token source that forwards all tokens from this token source that have none of the given types.
    default TokenSource<D,T>
    removeTypes(T... types)
    Returns a secondary token source that forwards all tokens from this token source that have none of the given types.
    default TokenSource<D,T>
    with(Function<? super TokenSource<D,T>,? extends TokenProcessor<D,T>> processor)
    Creates a new token processor by applying a given factory function to this source.

    Methods inherited from interface java.util.function.Supplier

    get

  • Method Details

    • with

      default TokenSource<D,T> with(Function<? super TokenSource<D,T>,? extends TokenProcessor<D,T>> processor)
      Creates a new token processor by applying a given factory function to this source.
      Parameters:
      processor - the factory function for creating a processor
      Returns:
      a new token processor by applying the given factory function to this source

    • removeTypes

      default TokenSource<D,T> removeTypes(T... types)
      Returns a secondary token source that forwards all tokens from this token source that have none of the given types.
      Parameters:
      types - the types of tokens to remove
      Returns:
      a token source that forwards all tokens from this token source that have none of the given types

    • removeTypes

      default TokenSource<D,T> removeTypes(Set<T> types)
      Returns a secondary token source that forwards all tokens from this token source that have none of the given types.
      Parameters:
      types - the types of tokens to remove
      Returns:
      a token source that forwards all tokens from this token source that have none of the given types