Enhanced C#
Language of your choice: library documentation
Public Member Functions | List of all members
Loyc.Syntax.Lexing.TokensToTree Class Reference

A preprocessor usually inserted between the lexer and parser that converts a token list into a token tree. Everything inside brackets, parens or braces is made a child of the open bracket. More...


Source file:
Inheritance diagram for Loyc.Syntax.Lexing.TokensToTree:
Loyc.Syntax.Lexing.LexerWrapper< Token > Loyc.Syntax.Lexing.ILexer< Token > Loyc.Syntax.IIndexToLine

Remarks

A preprocessor usually inserted between the lexer and parser that converts a token list into a token tree. Everything inside brackets, parens or braces is made a child of the open bracket.

The close bracketis not treated as one of the children of the opening bracket.

Public Member Functions

 TokensToTree (ILexer< Token > source, bool skipWhitespace)
 
override Maybe< TokenNextToken ()
 Returns the next (postprocessed) token. This method should set the _current field to the returned value. More...
 
- Public Member Functions inherited from Loyc.Syntax.Lexing.LexerWrapper< Token >
 LexerWrapper (ILexer< Token > sourceLexer)
 
SourcePos IndexToLine (int index)
 Returns the position in a source file of the specified index. More...
 
virtual void Reset ()
 

Additional Inherited Members

- Properties inherited from Loyc.Syntax.Lexing.LexerWrapper< Token >
ILexer< TokenLexer [get, set]
 
ISourceFile SourceFile [get]
 
virtual IMessageSink ErrorSink [get, set]
 
int IndentLevel [get]
 
UString IndentString [get]
 
int LineNumber [get]
 
int InputPosition [get]
 
string FileName [get]
 
- Properties inherited from Loyc.Syntax.Lexing.ILexer< Token >
ISourceFile SourceFile [get]
 The file being lexed. More...
 
IMessageSink ErrorSink [get, set]
 Event handler for errors. More...
 
int IndentLevel [get]
 Indentation level of the current line. This is updated after scanning the first whitespaces on a new line, and may be reset to zero when NextToken() returns a newline. More...
 
UString IndentString [get]
 Gets a string slice that holds the spaces or tabs that were used to indent the current line. More...
 
int LineNumber [get]
 Current line number (1 for the first line). More...
 
int InputPosition [get]
 Current input position (an index into SourceFile.Text). More...
 
- Properties inherited from Loyc.Syntax.IIndexToLine
string FileName [get]
 Gets the file name used in results returned by IndexToLine(int). More...
 
- Protected Member Functions inherited from Loyc.Syntax.Lexing.LexerWrapper< Token >
void WriteError (int index, string msg, params object[] args)
 
- Protected fields inherited from Loyc.Syntax.Lexing.LexerWrapper< Token >
Maybe< Token_current
 

Member Function Documentation

override Maybe<Token> Loyc.Syntax.Lexing.TokensToTree.NextToken ( )
inlinevirtual

Returns the next (postprocessed) token. This method should set the _current field to the returned value.

Implements Loyc.Syntax.Lexing.LexerWrapper< Token >.

References Loyc.Collections.DList< T >.Add(), and Loyc.Syntax.IndexPositionMapper< CharSource >.IndexToLine().