Remarks

Lexer for EC# source code.

See also: ILexer{Token}, TokensToTree

Public fields
bool	AllowNestedComments = true

bool	SkipValueParsing = false
	Used for syntax highlighting, which doesn't care about token values. This option causes the Token.Value to be set to a default, like '\0' for single-quoted strings and 0 for numbers. Operator names are still parsed. More...

Public Member Functions
	Les2Lexer (UString text, IMessageSink errorSink)

	Les2Lexer (ICharSource text, string fileName, IMessageSink sink, int startPosition=0)

override void	Reset (ICharSource source, string fileName="", int inputPosition=0, bool newSourceFile=true)

Les2Lexer	Clone ()

override Maybe< Token >	NextToken ()
	Scans the next token and returns information about it. More...

bool	TDQStringLine ()

bool	TSQStringLine ()

bool	MLCommentLine (ref int nested)

Public Member Functions inherited from Loyc.Syntax.IIndexToLine
ILineColumnFile	IndexToLine (int index)
	Returns the position in a source file of the specified index. More...

Static Public Member Functions
static string	UnescapeQuotedString (ref UString sourceText, Action< int, string > onError, UString indentation=default(UString), bool les3TQIndents=false)
	Parses a normal or triple-quoted string that still includes the quotes. Supports quote types '\'', '"' and '`'. More...

static void	UnescapeQuotedString (ref UString sourceText, Action< int, string > onError, StringBuilder sb, UString indentation=default(UString), bool les3TQIndents=false)
	Parses a normal or triple-quoted string that still includes the quotes (see documentation of the first overload) into a StringBuilder. More...

static bool	UnescapeString (ref UString sourceText, char quoteType, bool isTripleQuoted, Action< int, string > onError, StringBuilder sb, UString indentation=default(UString), bool les3TQIndents=false)
	Parses a normal or triple-quoted string whose starting quotes have been stripped out. If triple-quote parsing was requested, stops parsing at three quote marks; otherwise, stops parsing at a single end-quote or newline. More...

static string	ParseIdentifier (ref UString source, Action< int, string > onError, out bool checkForNamedLiteral)
	Parses an LES2-style identifier such as `foo`, , `<tt>foo` or `–punctuation–`. More...

Protected Member Functions
override void	Error (int lookaheadIndex, string message, params object[] args)

UString	Text ()

sealed override void	AfterNewline ()

override bool	SupportDotIndents ()

void	UnescapeSQStringValue ()

Symbol	ParseBQStringValue ()

UString	UnescapeString (bool isTripleQuoted, bool allowExtraIndent=false)

object	ParseIdValue (bool isFancy)

void	UnescapeSymbolValue ()

Symbol	IdToSymbol (UString ustr)

object	ParseNormalOp ()

Protected fields
bool	_isFloat

NodeStyle	_style

int	_numberBase

TokenType	_type

object	_value

UString	_textValue

int	_startPosition

Dictionary< UString, Symbol >	_idCache = new Dictionary<UString,Symbol>()

Additional Inherited Members
Properties inherited from Loyc.Syntax.Lexing.ILexer< Token >
ISourceFile	SourceFile `[get]`
	The file being lexed. More...

IMessageSink	ErrorSink `[get, set]`
	Event handler for errors. More...

int	IndentLevel `[get]`
	Indentation level of the current line. This is updated after scanning the first whitespaces on a new line, and may be reset to zero when NextToken() returns a newline. More...

UString	IndentString `[get]`
	Gets a string slice that holds the spaces or tabs that were used to indent the current line. More...

int	LineNumber `[get]`
	Current line number (1 for the first line). More...

int	InputPosition `[get]`
	Current input position (an index into SourceFile.Text). More...

Properties inherited from Loyc.Syntax.IFileName
string	FileName `[get]`

Member Function Documentation

override Maybe<Token> Loyc.Syntax.Les.Les2Lexer.NextToken ( )

inline

Scans the next token and returns information about it.

Returns: The next token, or null at the end of the source file.

Implements Loyc.Syntax.Lexing.ILexer< Token >.

References Loyc.Newline, Loyc.Syntax.Operator, and Loyc.Syntax.Lexing.Spaces.

static string Loyc.Syntax.Les.Les2Lexer.ParseIdentifier	(	ref UString	source,
		Action< int, string >	onError,
		out bool	checkForNamedLiteral
	)

inlinestatic

Parses an LES2-style identifier such as foo, , <tt>foo or –punctuation–.

Parameters

source	Text to parse. On return, the range has been decreased by the length of the token; this method also stops if this range becomes empty.
onError	A method to call on error
checkForNamedLiteral	This is set to true when the input starts with @ but doesn't use backquotes, which could indicate that it is an LES named literal such as or .

Returns: The parsed version of the identifier.

static string Loyc.Syntax.Les.Les2Lexer.UnescapeQuotedString	(	ref UString	sourceText,
		Action< int, string >	onError,
		UString	indentation = `default(UString)`,
		bool	les3TQIndents = `false`
	)

static

Parses a normal or triple-quoted string that still includes the quotes. Supports quote types '\'', '"' and '`'.

Parameters

sourceText	input text
onError	Called in case of parsing error (unknown escape sequence or missing end quotes)
indentation	Inside a triple-quoted string, any text following a newline is ignored as long as it matches this string. For example, if the text following a newline is "\t\t Foo" and this string is "\t\t\t", the tabs are ignored and " Foo" is kept.
les3TQIndents	Enable EC# triple-quoted string indent rules, which allow an additional one tab or three spaces of indent. (I'm leaning toward also supporting this in LES; switched on in v3)

Returns: The decoded string

This method recognizes LES and EC#-style string syntax. Firstly, it recognizes triple-quoted strings (''' """ ```). These strings enjoy special newline handling: the newline is always interpreted as
regardless of the actual kind of newline ( and
newlines come out as
), and indentation following the newline can be stripped out. Triple-quoted strings can have escape sequences that use both kinds of slash, like so: / / \'/ "/ \0/. However, there are no unicode escapes (/ is NOT supported).

Secondly, it recognizes normal strings (' " `). These strings stop parsing (with an error) at a newline, and can contain C-style escape sequences: \' " \0 etc. C#-style verbatim strings are NOT supported.

static void Loyc.Syntax.Les.Les2Lexer.UnescapeQuotedString	(	ref UString	sourceText,
		Action< int, string >	onError,
		StringBuilder	sb,
		UString	indentation = `default(UString)`,
		bool	les3TQIndents = `false`
	)

static

Parses a normal or triple-quoted string that still includes the quotes (see documentation of the first overload) into a StringBuilder.

static bool Loyc.Syntax.Les.Les2Lexer.UnescapeString	(	ref UString	sourceText,
		char	quoteType,
		bool	isTripleQuoted,
		Action< int, string >	onError,
		StringBuilder	sb,
		UString	indentation = `default(UString)`,
		bool	les3TQIndents = `false`
	)

static

Parses a normal or triple-quoted string whose starting quotes have been stripped out. If triple-quote parsing was requested, stops parsing at three quote marks; otherwise, stops parsing at a single end-quote or newline.

Returns: true if parsing stopped at one or three quote marks, or false if parsing stopped at the end of the input string or at a newline (in a string that is not triple-quoted).

This method recognizes LES and EC#-style string syntax.

Member Data Documentation

bool Loyc.Syntax.Les.Les2Lexer.SkipValueParsing = false

Used for syntax highlighting, which doesn't care about token values. This option causes the Token.Value to be set to a default, like '\0' for single-quoted strings and 0 for numbers. Operator names are still parsed.

Remarks

Public fields

Public Member Functions

Static Public Member Functions

Protected Member Functions

Protected fields

Additional Inherited Members

Member Function Documentation

Member Data Documentation