Package eu.bandm.tools.lexic

Class LookaheadTokenFilter<D,T,L>

java.lang.Object
eu.bandm.tools.lexic.LookaheadTokenProcessor<D,T,L>
eu.bandm.tools.lexic.LookaheadTokenFilter<D,T,L>
Type Parameters:
D - the type of source document identifiers
T - the type of token types
L - the type of lookahead data
All Implemented Interfaces:
LookaheadTokenSource<D,T,L>, Supplier<Token<D,T>>
Direct Known Subclasses:
TokenFilter
public class LookaheadTokenFilter<D,T,L> extends LookaheadTokenProcessor<D,T,L>
Abstract base class for token processors that filter out certain tokens. Tokens from the given input token source are either forwarded or discarded, depending on a variety of criteria.

Note that the incoming token source is expected to supply an infinite number of special tokens that indicate the end of the input stream. Attempting to filter out these tokens will result in an infinite loop. Combine only token sources and filters that have compatible application-specific conventions on how the end of the input stream is tokenized.

  • Constructor Details

    • LookaheadTokenFilter

      protected LookaheadTokenFilter(LookaheadTokenSource<D,T,L> input, Predicate<? super Token<D,T>> acceptable)
      Creates a new instance.
      Parameters:
      input - the input token source
      acceptable - a function that yields true for acceptable tokens, and false for those to be filtered out

  • Method Details

    • get

      public Token<D,T> get()

      This implementation returns the next acceptable token from the input source; intervening tokens may be discarded silently.

    • forward

      public static <D, T, L> LookaheadTokenFilter<D,T,L> forward(LookaheadTokenSource<D,T,L> input, Predicate<? super Token<D,T>> pred)
      Returns a token filter that forwards those tokens from a given input source that match a given predicate.

      End-of-stream tokens should always be forwarded.

      Type Parameters:
      D - the type of source document identifiers
      T - the type of token types
      L - the type of lookahead data
      Parameters:
      input - the input token source
      pred - the predicate to match
      Returns:
      a token filter that forwards those tokens from the given input source that match the given predicate

    • discard

      public static <D, T, L> LookaheadTokenFilter<D,T,L> discard(LookaheadTokenSource<D,T,L> input, Predicate<? super Token<D,T>> pred)
      Returns a token filter that forwards only those tokens from a given input source that do not match a given predicate.

      End-of-stream tokens should always be forwarded.

      Type Parameters:
      D - the type of source document identifiers
      T - the type of token types
      L - the type of lookahead data
      Parameters:
      input - the input token source
      pred - the predicate not to match
      Returns:
      a token filter that forwards those tokens from the given input source that do not match the given predicate

    • removeTypes

      public static <D, T, L> LookaheadTokenFilter<D,T,L> removeTypes(LookaheadTokenSource<D,T,L> input, Set<? extends T> types)
      Returns a secondary token source that forwards all tokens from a given token source that have none of the given types.

      End-of-stream tokens should always be forwarded.

      Type Parameters:
      D - the type of source document identifiers
      T - the type of token types
      L - the type of lookahead data
      Parameters:
      input - the input token source
      types - the types of tokens to remove
      Returns:
      a token source that forwards all tokens from the given token source that have none of the given types

    • removeTypes

      @SafeVarargs public static <D, T, L> LookaheadTokenFilter<D,T,L> removeTypes(LookaheadTokenSource<D,T,L> input, T... types)
      Returns a secondary token source that forwards all tokens from a given token source that have none of the given types.

      End-of-stream tokens should always be forwarded.

      Type Parameters:
      D - the type of source document identifiers
      T - the type of token types
      L - the type of lookahead data
      Parameters:
      input - the input token source
      types - the types of tokens to remove
      Returns:
      a token source that forwards all tokens from the given token source that have none of the given types