Package eu.bandm.tools.lexic

Class TokenFilter<D,T>

java.lang.Object
eu.bandm.tools.lexic.TokenProcessor<D,T>
eu.bandm.tools.lexic.TokenFilter<D,T>
Type Parameters:
D - the type of source document identifiers
T - the type of token types
All Implemented Interfaces:
TokenSource<D,T>, Supplier<Token<D,T>>
public abstract class TokenFilter<D,T> extends TokenProcessor<D,T>
Abstract base class for token processors that filter out certain tokens. Tokens from the given input token source are either forwarded or discarded, depending on a variety of criteria.

  • Constructor Details

    • TokenFilter

      protected TokenFilter(TokenSource<D,T> input)
      Creates a new instance.
      Parameters:
      input - the input token source

  • Method Details

    • get

      public Token<D,T> get()

      This implementation returns the next acceptable token from the input source; intervening tokens may be discarded silently.

    • acceptable

      protected abstract boolean acceptable(Token<D,T> token)
      Checks whether a given token is to be forwarded or discarded.
      Parameters:
      token - the token
      Returns:
      true if the token is to be forwarded; false if it is to be discarded

    • forward

      public static <D, T> TokenFilter<D,T> forward(TokenSource<D,T> input, Predicate<? super Token<D,T>> pred)
      Returns a token filter that forwards those tokens from a given input source that match a given predicate.
      Type Parameters:
      D - the type of source document identifiers
      T - the type of token types
      Parameters:
      input - the input token source
      pred - the predicate to match
      Returns:
      a token filter that forwards those tokens from the given input source that match the given predicate

    • discard

      public static <D, T> TokenFilter<D,T> discard(TokenSource<D,T> input, Predicate<? super Token<D,T>> pred)
      Returns a token filter that forwards those tokens from a given input source that do not match a given predicate.
      Type Parameters:
      D - the type of source document identifiers
      T - the type of token types
      Parameters:
      input - the input token source
      pred - the predicate not to match
      Returns:
      a token filter that forwards those tokens from the given input source that do not match the given predicate

    • removeTypes

      public static <D, T> TokenFilter<D,T> removeTypes(TokenSource<D,T> input, Set<T> types)
      Returns a secondary token source that forwards all tokens from a given token source that have none of the given types.
      Type Parameters:
      D - the type of source document identifiers
      T - the type of token types
      Parameters:
      input - the input token source
      types - the types of tokens to remove
      Returns:
      a token source that forwards all tokens from the given token source that have none of the given types