A preprocessor usually inserted between the lexer and parser that converts a token list into a token tree. Everything inside brackets, parens or braces is made a child of the open bracket. More...
Source file:
Inheritance diagram for Loyc.Syntax.Lexing.TokensToTree:
Remarks
A preprocessor usually inserted between the lexer and parser that converts a token list into a token tree. Everything inside brackets, parens or braces is made a child of the open bracket.
The close bracketis not treated as one of the children of the opening bracket.
Public Member Functions | |
| TokensToTree (ILexer< Token > source, bool skipWhitespace) | |
| override Maybe< Token > | NextToken () |
Returns the next (postprocessed) token. This method should set the _current field to the returned value. More... | |
 Public Member Functions inherited from Loyc.Syntax.Lexing.LexerWrapper< Token > | |
| LexerWrapper (ILexer< Token > sourceLexer) | |
| ILineColumnFile | IndexToLine (int index) |
| Returns the position in a source file of the specified index. More... | |
| virtual void | Reset () |
Additional Inherited Members | |
| Properties inherited from Loyc.Syntax.Lexing.LexerWrapper< Token > | |
| ILexer< Token > | Lexer [get, set] |
| ISourceFile | SourceFile [get] |
| virtual IMessageSink | ErrorSink [get, set] |
| int | IndentLevel [get] |
| UString | IndentString [get] |
| int | LineNumber [get] |
| int | InputPosition [get] |
| string | FileName [get] |
| Properties inherited from Loyc.Syntax.Lexing.ILexer< Token > | |
| ISourceFile | SourceFile [get] |
| The file being lexed. More... | |
| IMessageSink | ErrorSink [get, set] |
| Event handler for errors. More... | |
| int | IndentLevel [get] |
| Indentation level of the current line. This is updated after scanning the first whitespaces on a new line, and may be reset to zero when NextToken() returns a newline. More... | |
| UString | IndentString [get] |
| Gets a string slice that holds the spaces or tabs that were used to indent the current line. More... | |
| int | LineNumber [get] |
| Current line number (1 for the first line). More... | |
| int | InputPosition [get] |
| Current input position (an index into SourceFile.Text). More... | |
| Properties inherited from Loyc.Syntax.IHasFileName | |
| string | FileName [get] |
| Protected Member Functions inherited from Loyc.Syntax.Lexing.LexerWrapper< Token > | |
| void | WriteError (int index, string msg, params object[] args) |
| Protected fields inherited from Loyc.Syntax.Lexing.LexerWrapper< Token > | |
| Maybe< Token > | _current |
Member Function Documentation
◆ NextToken()
Returns the next (postprocessed) token. This method should set the _current field to the returned value.
Implements Loyc.Syntax.Lexing.LexerWrapper< Token >.
