Package eu.bandm.tools.lexic

Class TokenFilter<D,T>

java.lang.Object
eu.bandm.tools.lexic.LookaheadTokenProcessor<D,T,Void>
eu.bandm.tools.lexic.LookaheadTokenFilter<D,T,Void>
eu.bandm.tools.lexic.TokenFilter<D,T>
Type Parameters:
D - the type of source document identifiers
T - the type of token types
All Implemented Interfaces:
LookaheadTokenSource<D,T,Void>, TokenSource<D,T>, Closeable, AutoCloseable, Supplier<Token<D,T>>
public class TokenFilter<D,T> extends LookaheadTokenFilter<D,T,Void> implements TokenSource<D,T>
Abstract base class for token processors that filter out certain tokens. Tokens from the given input token source are either forwarded or discarded, depending on a variety of criteria.

  • Constructor Details

    • TokenFilter

      protected TokenFilter(TokenSource<D,T> input, Predicate<? super Token<D,T>> acceptable)
      Creates a new instance.
      Parameters:
      input - the input token source
      acceptable - a predicte that returns true for tokens that should be retained, false for tokens that should be discarded

  • Method Details

    • discard

      public static <D, T> TokenFilter<D,T> discard(TokenSource<D,T> input, Predicate<? super Token<D,T>> pred)
      Returns a token filter that forwards only those tokens from a given input source that do not match a given predicate.

      End-of-stream tokens should always be forwarded.

      Type Parameters:
      D - the type of source document identifiers
      T - the type of token types
      Parameters:
      input - the input token source
      pred - the predicate not to match
      Returns:
      a token filter that forwards those tokens from the given input source that do not match the given predicate

    • removeTypes

      public static <D, T> TokenFilter<D,T> removeTypes(TokenSource<D,T> input, Set<? extends T> types)
      Returns a secondary token source that forwards all tokens from a given token source that have none of the given types.

      End-of-stream tokens should always be forwarded.

      Type Parameters:
      D - the type of source document identifiers
      T - the type of token types
      Parameters:
      input - the input token source
      types - the types of tokens to remove
      Returns:
      a token source that forwards all tokens from the given token source that have none of the given types

    • removeTypes

      @SafeVarargs public static <D, T> TokenFilter<D,T> removeTypes(TokenSource<D,T> input, T... types)
      Returns a secondary token source that forwards all tokens from a given token source that have none of the given types.

      End-of-stream tokens should always be forwarded.

      Type Parameters:
      D - the type of source document identifiers
      T - the type of token types
      Parameters:
      input - the input token source
      types - the types of tokens to remove
      Returns:
      a token source that forwards all tokens from the given token source that have none of the given types