MemScanner (BandM metatools API doc)

java.lang.Object
- eu.bandm.tools.d2d2.infra.MemString
- - eu.bandm.tools.d2d2.base.MemScanner

All Implemented Interfaces:

Cloneable
```
public class MemScanner
extends MemString
```
Basic scanner for d2d text recognition, working on and returning MemString objects.

During work, the "result" field lastTokenType, lastContent, etc. describe the last recognized input token. This extends up to MemString.start, which is the "next-to-decode" reading position.

Each call to accept() means that the current result info has been consumed by the calling layer. So it is forgotten, the next token is decoded at the next-to-read adress, this address is advanced, and all result fields are updated.

Secondly, there are character-wise "back door entries" needed for character level parsing. These DUPLICATE parts of the "accept()" code.

Nested Class Summary

Nested Classes
Modifier and Type Class and Description

static class MemScanner.OpenTagModifier

static class MemScanner.TokenType
- Nested classes/interfaces inherited from class eu.bandm.tools.d2d2.infra.MemString
  MemString.CharMem, MemString.ReadBehindBufferEndException, MemString.Reader

Nested Classes
Modifier and Type	Class and Description
`static class`	`MemScanner.OpenTagModifier`
`static class`	`MemScanner.TokenType`

Field Summary

Fields
Modifier and Type	Field and Description
`protected String`	`closedTag` output parameter, full-text tag in case of lastTokenType=closeChar
`protected char`	`commandchar` Local state which can be altered by the caller.
`protected char`	`commentchar_1` Local state which can be altered by the caller.
`protected String`	`commentstring_ml`
`protected String`	`commentstring_sl`
`protected Character`	`current_toplevel_end` cache for top-most closing parenthesis, for performance only.
`protected Stack<Location<String>>`	`endlocations` internal state, holding the positions of the paren's activation.
`protected Stack<Character>`	`endparentheses` internal state, holding the currently expected parentheses
`protected Stack<String>`	`endtags` internal state, holding the full text tags corresponding to parenths.
`protected MemString`	`lastContent` output parameter, holding the character content consumed as last
`protected int`	`lastNumericValue` output parameter
`protected MemScanner.TokenType`	`lastTokenType` output parameter
`static CharSet`	`nonBlank`

Fields inherited from class eu.bandm.tools.d2d2.infra.MemString
CHAR_EOTEXT, CHAR_NEWLINE, colnum, data, len, linenum, start

Constructor Summary

Constructors
Constructor and Description

MemScanner(MemString.CharMem m)

MemScanner(MemString m)

Constructors
Constructor and Description
`MemScanner(MemString.CharMem m)`
`MemScanner(MemString m)`

Method Summary

All Methods Instance Methods Concrete Methods
Modifier and Type	Method and Description
`void`	`accept_blanks_filtered()` Called from `CharacterParser` and from `Text2Udom` ONCE before character and enumeration parsing starts.
`void`	`accept_blanks_noCommentsAllowed()` commandchar, commentchar, eof and parenthesis chars are never contained in any def of "whitespace characters".
`int`	`accept_greedy_filtered_chars(CharSet set)` Only called (only once) from `CharacterParser`.
`boolean`	`accept_one_filtered_char(CharSet set)` Only called from CharacterParser.
`boolean`	`accept_one(char c)` Only called from `Text2Udom` when decoding file header.
`boolean`	`accept_one(CharSet cs)`
`boolean`	`accept_string_w_o_lineswitch(String s)` Assume String s does NOT contain any line switching characters! FIXME this must be checked in char parser type checker of STRING The string is accepted if it is continuously contained outside of comment, AND does not contain the current top-level close paranethesis character. The command char is currently not rejected, i.e.
`void`	`accept()` Central scanning procedure: (1) throws away old results, (2) consumes more characters and (3) sets `get_lastTokenType()`, `last_content`, `get_lastLocation()` etc.
`MemScanner`	`copy()` returns a clone.
`Character`	`endTagCharacter(String tag)` Returns the currently expected first "close parenthesis" character.
`MemString`	`get_lastContent()` interface procedure: deliver MemString lastly consumed.
`Location<String>`	`get_lastLocation()` interface procedure: character location lastly consumed.
`MemScanner.TokenType`	`get_lastTokenType()` interface procedure: deliver token type of last consumption act.
`int`	`get_numericValue()` interface procedure: deliver numeric tag (for direct unicode access!).
`char`	`getCommandChar()`
`char`	`getCommentChar_1()` "comment char 2" is fixed to "*" in `Chars.CHAR_COMMENT_2`.
`Location<String>`	`getParenthesisDefinitionLocation(String tag)` ONLY called from warning for pending/possibly unused closed par!
`MemScanner.OpenTagModifier`	`markParenthesis(String tag)` consume an open parenthesis and memoize its counterpart.
`Location<String>`	`popParenthesis()` May only be called when a close-par has been signalled as recognized!
`String`	`read_ident_or_null()` Only called from `Text2Udom` when decoding file header.
`String`	`read_nonblank()` Only called from `Text2Udom` when decoding file header.
`char`	`scan_again_skipping_first()` HACK: reset state of parser, as it was before the last "accept", but skip the first character of the last recognized result.
`void`	`setCommandChar(char c)` Interface procedure: change command char.
`void`	`setCommentChar_1(char c)` Interface procedure: change comment char.
`void`	`skip_after_char(char c)` ASSUMEs that character is really present.
`void`	`skip_until_eoline()` Is called from Text2Udom digest_nothing_open() and digest_consume_characters(), when a "`///`" = "incomplete close" token has been consumed in the role of a comment start.

Methods inherited from class eu.bandm.tools.d2d2.infra.MemString
adjustLineNumberX, advance, advanceBy, advanceByX_sameLine, advanceX, appendAll, asString, backwardsX, backwardsX, calccolnum, clearLineX, equals, get_colnum, get_data, get_len, get_linenum, get_start, getChar, getChar, getFollowLocation, getLocation, getNextLine, getNextLineX, getPositionIndication, initFrom, isPrefixedBy, main, putChar, toString, upTo, upToX

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

- Field Detail
  - lastTokenType
```
protected MemScanner.TokenType lastTokenType
```
    output parameter
  - lastNumericValue
```
protected int lastNumericValue
```
    output parameter
  - lastContent
```
protected MemString lastContent
```
    output parameter, holding the character content consumed as last
  - closedTag
```
protected String closedTag
```
    output parameter, full-text tag in case of lastTokenType=closeChar
  - endparentheses
```
protected Stack<Character> endparentheses
```
    internal state, holding the currently expected parentheses
  - endtags
```
protected Stack<String> endtags
```
    internal state, holding the full text tags corresponding to parenths.
  - endlocations
```
protected Stack<Location<String>> endlocations
```
    internal state, holding the positions of the paren's activation.
  - current_toplevel_end
```
protected Character current_toplevel_end
```
    cache for top-most closing parenthesis, for performance only.
  - commandchar
```
protected char commandchar
```
    Local state which can be altered by the caller.
  - commentchar_1
```
protected char commentchar_1
```
    Local state which can be altered by the caller. "comment char 2" is fixed to "*" in Chars.CHAR_COMMENT_2.
  - commentstring_sl
```
protected String commentstring_sl
```
  - commentstring_ml
```
protected String commentstring_ml
```
  - nonBlank
```
public static final CharSet nonBlank
```
- Constructor Detail
  - MemScanner
```
public MemScanner(MemString m)
```
  - MemScanner
```
public MemScanner(MemString.CharMem m)
```
- Method Detail
  - copy
```
public MemScanner copy()
```
    Description copied from class: MemString
    
    returns a clone.
    
    Overrides:
    
    copy in class MemString
  - markParenthesis
```
public MemScanner.OpenTagModifier markParenthesis(String tag)
```
    consume an open parenthesis and memoize its counterpart. Must ALWAYS be called whenever an open tag is recognized (by the code of the higher level), because it consumes the trailing blank char. FIXME warum nicht HIER fest dahinter ?? It must be called before the corresponding call to accept().. Consumes one(1) char if it is an open-par, and memoizes the closing one. Consumes one(1) char if it is a whitespace !
```
        #  tag CONTINUE TEXT
        #  tag/CONTINUE TEXT
        #  tag///CONTINUE TEXT
        #  tag//comment recognizes as such(incl lead in!)
        #  tag
        CONTINUE TEXT
        #  tag(continue text with parenthesis open 
        #  tag=continue text including leading special, non-paren char 
        #  tag()xx  same as #tag/xx
        
```
  - popParenthesis
```
public Location<String> popParenthesis()
```
    May only be called when a close-par has been signalled as recognized!
  - endTagCharacter
```
public Character endTagCharacter(String tag)
```
    Returns the currently expected first "close parenthesis" character.
  - getParenthesisDefinitionLocation
```
public Location<String> getParenthesisDefinitionLocation(String tag)
```
    ONLY called from warning for pending/possibly unused closed par!
  - setCommandChar
```
public void setCommandChar(char c)
```
    Interface procedure: change command char. Attention no checking, caller is fully responsible.
  - setCommentChar_1
```
public void setCommentChar_1(char c)
```
    Interface procedure: change comment char. Attention no checking, caller is fully responsible.
  - getCommandChar
```
public char getCommandChar()
```
  - getCommentChar_1
```
public char getCommentChar_1()
```
    "comment char 2" is fixed to "*" in Chars.CHAR_COMMENT_2.
  - get_lastTokenType
```
public MemScanner.TokenType get_lastTokenType()
```
    interface procedure: deliver token type of last consumption act.
  - get_lastContent
```
public MemString get_lastContent()
```
    interface procedure: deliver MemString lastly consumed.
  - get_numericValue
```
public int get_numericValue()
```
    interface procedure: deliver numeric tag (for direct unicode access!).
  - get_lastLocation
```
public Location<String> get_lastLocation()
```
    interface procedure: character location lastly consumed.
  - accept
```
public void accept()
```
    Central scanning procedure: (1) throws away old results, (2) consumes more characters and (3) sets get_lastTokenType(), last_content, get_lastLocation() etc. accordingly. Never fails.
    
    PROBLEM of priorities:
```
                 "///" --> forced end quote      
   otherwise     "//" --> one-line comment start (if comment char has not been altered.)
   otherwise     "/" --> end quote
   
```
    NOTES:
    command char can never be set to "/" (nor to current comment)
    comment lead-in char can never be set to "*" (nor to current command)
  - scan_again_skipping_first
```
public char scan_again_skipping_first()
```
    HACK: reset state of parser, as it was before the last "accept", but skip the first character of the last recognized result. This is needed for Text2Udom#commandchar() and Text2Udom#commentchar(), otherwise the standard "accept()" interface would become much more complicated !
  - skip_until_eoline
```
public void skip_until_eoline()
```
    Is called from Text2Udom digest_nothing_open() and digest_consume_characters(), when a "///" = "incomplete close" token has been consumed in the role of a comment start.
  - skip_after_char
```
public void skip_after_char(char c)
```
    ASSUMEs that character is really present. Otherwise it throws a ReadBehindBufferEndException. Only called (once) in Text2Udom, when parsing header.
  - accept_greedy_filtered_chars
```
public int accept_greedy_filtered_chars(CharSet set)
```
    Only called (only once) from CharacterParser. Comments and parenthesis chars have priority over character parsers. The current command character is accepted when in the parser def the standard command character appears.
  - accept_one_filtered_char
```
public boolean accept_one_filtered_char(CharSet set)
```
    Only called from CharacterParser.
  - accept_blanks_filtered
```
public void accept_blanks_filtered()
```
    Called from CharacterParser and from Text2Udom ONCE before character and enumeration parsing starts.
  - accept_blanks_noCommentsAllowed
```
public void accept_blanks_noCommentsAllowed()
```
    commandchar, commentchar, eof and parenthesis chars are never contained in any def of "whitespace characters". Only called from Text2Udom when decoding file header. There no comments may appear.
  - read_ident_or_null
```
public String read_ident_or_null()
```
    Only called from Text2Udom when decoding file header.
  - read_nonblank
```
public String read_nonblank()
```
    Only called from Text2Udom when decoding file header.
  - accept_one
```
public boolean accept_one(char c)
```
    Only called from Text2Udom when decoding file header. (comments are not allowed there) DOCU FIXME)
  - accept_one
```
public boolean accept_one(CharSet cs)
```
  - accept_string_w_o_lineswitch
```
public boolean accept_string_w_o_lineswitch(String s)
```
    Assume String s does NOT contain any line switching characters!
    FIXME this must be checked in char parser type checker of STRING
    
    The string is accepted if it is continuously contained outside of comment, AND does not contain the current top-level close paranethesis character.
    The command char is currently not rejected, i.e. may be contained in the argument and will not be translated.(FIXME??)
    Only called by CharacterParser for "String" expressions, by Text2Udom when parsing the file header, and when parsing an Enumeration value.

Class MemScanner

Nested Class Summary

Nested classes/interfaces inherited from class eu.bandm.tools.d2d2.infra.MemString

Field Summary

Fields inherited from class eu.bandm.tools.d2d2.infra.MemString

Constructor Summary

Method Summary

Methods inherited from class eu.bandm.tools.d2d2.infra.MemString

Methods inherited from class java.lang.Object

Field Detail

lastTokenType

lastNumericValue

lastContent

closedTag

endparentheses

endtags

endlocations

current_toplevel_end

commandchar

commentchar_1

commentstring_sl

commentstring_ml

nonBlank

Constructor Detail

MemScanner

MemScanner

Method Detail

copy

markParenthesis

popParenthesis

endTagCharacter

getParenthesisDefinitionLocation

setCommandChar

setCommentChar_1

getCommandChar

getCommentChar_1

get_lastTokenType

get_lastContent

get_numericValue

get_lastLocation

accept

scan_again_skipping_first

skip_until_eoline

skip_after_char

accept_greedy_filtered_chars

accept_one_filtered_char

accept_blanks_filtered

accept_blanks_noCommentsAllowed

read_ident_or_null

read_nonblank

accept_one

accept_one

accept_string_w_o_lineswitch