Package eu.bandm.tools.lexic

Class LookaheadTokenFilter<D,T,L>

java.lang.Object
eu.bandm.tools.lexic.LookaheadTokenProcessor<D,T,L>
eu.bandm.tools.lexic.LookaheadTokenFilter<D,T,L>
Type Parameters:
D - the type of source document identifiers
T - the type of token types
All Implemented Interfaces:
LookaheadTokenSource<D,T,L>, Supplier<Token<D,T>>
Direct Known Subclasses:
TokenFilter
public class LookaheadTokenFilter<D,T,L> extends LookaheadTokenProcessor<D,T,L>
Abstract base class for token processors that filter out certain tokens. Tokens from the given input token source are either forwarded or discarded, depending on a variety of criteria.

  • Constructor Details

    • LookaheadTokenFilter

      protected LookaheadTokenFilter(LookaheadTokenSource<D,T,L> input, Predicate<? super Token<D,T>> acceptable)
      Creates a new instance.
      Parameters:
      input - the input token source

  • Method Details

    • get

      public Token<D,T> get()

      This implementation returns the next acceptable token from the input source; intervening tokens may be discarded silently.

    • forward

      public static <D, T, L> LookaheadTokenFilter<D,T,L> forward(LookaheadTokenSource<D,T,L> input, Predicate<? super Token<D,T>> pred)
      Returns a token filter that forwards those tokens from a given input source that match a given predicate.
      Type Parameters:
      D - the type of source document identifiers
      T - the type of token types
      Parameters:
      input - the input token source
      pred - the predicate to match
      Returns:
      a token filter that forwards those tokens from the given input source that match the given predicate

    • discard

      public static <D, T, L> LookaheadTokenFilter<D,T,L> discard(LookaheadTokenSource<D,T,L> input, Predicate<? super Token<D,T>> pred)
      Returns a token filter that forwards those tokens from a given input source that do not match a given predicate.
      Type Parameters:
      D - the type of source document identifiers
      T - the type of token types
      Parameters:
      input - the input token source
      pred - the predicate not to match
      Returns:
      a token filter that forwards those tokens from the given input source that do not match the given predicate

    • removeTypes

      public static <D, T, L> LookaheadTokenFilter<D,T,L> removeTypes(LookaheadTokenSource<D,T,L> input, Set<? extends T> types)
      Returns a secondary token source that forwards all tokens from a given token source that have none of the given types.
      Type Parameters:
      D - the type of source document identifiers
      T - the type of token types
      Parameters:
      input - the input token source
      types - the types of tokens to remove
      Returns:
      a token source that forwards all tokens from the given token source that have none of the given types