Remarks

A standard interface for lexers.

Template Parameters

Token Type of tokens produced by the lexer (usually Loyc.Syntax.Lexing.Token).

Properties
ISourceFile	SourceFile `[get]`
	The file being lexed. More...

IMessageSink	ErrorSink `[get, set]`
	Event handler for errors. More...

int	IndentLevel `[get]`
	Indentation level of the current line. This is updated after scanning the first whitespaces on a new line, and may be reset to zero when NextToken() returns a newline. More...

UString	IndentString `[get]`
	Gets a string slice that holds the spaces or tabs that were used to indent the current line. More...

int	LineNumber `[get]`
	Current line number (1 for the first line). More...

int	InputPosition `[get]`
	Current input position (an index into SourceFile.Text). More...

Properties inherited from Loyc.Syntax.IHasFileName
string	FileName `[get]`

Public Member Functions
Maybe< Token >	NextToken ()
	Scans the next token and returns information about it. More...

Public Member Functions inherited from Loyc.Syntax.IIndexToLine
ILineColumnFile	IndexToLine (int index)
	Returns the position in a source file of the specified index. More...

Member Function Documentation

◆ NextToken()

Maybe<Token> Loyc.Syntax.Lexing.ILexer< Token >.NextToken ( )

Scans the next token and returns information about it.

Returns: The next token, or null at the end of the source file.

Implemented in Loyc.Syntax.Les.Les3Lexer, Loyc.Syntax.Les.Les2Lexer, Loyc.Syntax.Lexing.IndentTokenGenerator< Token >, Loyc.Syntax.Lexing.BaseILexer< CharSrc, Token >, Loyc.Ecs.Parser.EcsPreprocessor, Loyc.Ecs.Parser.EcsLexer, Loyc.Syntax.Lexing.LexerWrapper< Token >, Loyc.Syntax.Lexing.TokensToTree, Loyc.Syntax.Lexing.TokenListAsLexer, Loyc.Syntax.Lexing.TriviaSaver, and Loyc.Syntax.Lexing.WhitespaceFilter< Token >.

Property Documentation

◆ ErrorSink

IMessageSink Loyc.Syntax.Lexing.ILexer< Token >.ErrorSink

getset

Event handler for errors.

◆ IndentLevel

int Loyc.Syntax.Lexing.ILexer< Token >.IndentLevel

get

Indentation level of the current line. This is updated after scanning the first whitespaces on a new line, and may be reset to zero when NextToken() returns a newline.