Package eu.bandm.tscore.base

Class Lexer

java.lang.Object

eu.bandm.tscore.base.Lexer

public class Lexer
extends Object

Definition for the basic tokenization of tscore input. The fundamental "raw" layer makes only rough separation of input-text components---the semantic oriented parsing happens in later processing steps. Assume that the input is free of chars < 0x20, except \n, \t, and \f, as e.g. guaranteed by LocationCodePointSource.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static enum	Lexer.TokenType

Method Summary

Modifier and Type	Method	Description
static <D> TokenSource<D,Lexer.TokenType>	tokenSource(Reader reader, D sourceIndication)	Constructs a TokenSource which parses the Reader according to the static lexer rules in this class.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Details

tokenSource

public static <D> TokenSource<D,Lexer.TokenType> tokenSource(Reader reader,
 D sourceIndication)

Constructs a TokenSource which parses the Reader according to the static lexer rules in this class. No token types are discarded, but all are propagated to the consumer. Error message locations are calculated based on Location.EMACS_FIRST_LINE_NUMBER and Location.EMACS_FIRST_COLUMN_NUMBER. No explicit messaging of input stream errors, but one token with

invalid reference

TokenType.ERROR

type.