Source files:

Inheritance diagram for Loyc.Syntax.Les.Les3Lexer:

Properties
bool	PreferCustomLiterals `[get, set]`
	If this flag is true, all literals except plain strings and true/false/null are stored as CustomLiteral, bypassing number parsing so that all original characters are preserved if the output is written back to text. More...

Properties inherited from Loyc.Syntax.Lexing.ILexer< Token >
ISourceFile	SourceFile `[get]`
	The file being lexed. More...

IMessageSink	ErrorSink `[get, set]`
	Event handler for errors. More...

int	IndentLevel `[get]`
	Indentation level of the current line. This is updated after scanning the first whitespaces on a new line, and may be reset to zero when NextToken() returns a newline. More...

UString	IndentString `[get]`
	Gets a string slice that holds the spaces or tabs that were used to indent the current line. More...

int	LineNumber `[get]`
	Current line number (1 for the first line). More...

int	InputPosition `[get]`
	Current input position (an index into SourceFile.Text). More...

Properties inherited from Loyc.Syntax.IFileName
string	FileName `[get]`

Public Member Functions
	Les3Lexer (string text, string fileName="")

	Les3Lexer (ICharSource text, string fileName, IMessageSink sink, int startPosition=0)

override Maybe< Token >	NextToken ()
	Scans the next token and returns information about it. More...

bool	TDQStringLine ()

bool	TSQStringLine ()

bool	MLCommentLine (ref int nested)

Public Member Functions inherited from Loyc.Syntax.IIndexToLine
ILineColumnFile	IndexToLine (int index)
	Returns the position in a source file of the specified index. More...

Static Public Member Functions
static object	ParseLiteral (Symbol typeMarker, UString unescapedText, out string?syntaxError)

static string	UnescapeQuotedString (ref UString sourceText, Action< int, string > onError, UString indentation=default(UString), bool allowExtraIndent=false, bool parseAsBytes=true)
	Parses a normal or triple-quoted string that still includes the quotes. Supports quote types '\'', '"' and '`'. More...

static void	UnescapeQuotedString (ref UString sourceText, Action< int, string > onError, StringBuilder sb, UString indentation=default(UString), bool allowExtraIndent=false, bool parseAsBytes=true)
	Parses a normal or triple-quoted string that still includes the quotes (see documentation of the first overload) into a StringBuilder. More...

static bool	UnescapeString (ref UString sourceText, char quoteType, bool isTripleQuoted, Action< int, string > onError, StringBuilder sb, UString indentation=default(UString), bool allowExtraIndent=false, bool parseAsBytes=true)
	Parses a normal or triple-quoted string whose starting quotes have been stripped out. If triple-quote parsing was requested, stops parsing at three quote marks; otherwise, stops parsing at a single end-quote or newline. More...

static TokenType	GetOperatorTokenType (UString op)
	Under the assumption that `op` is a sequence of punctuation marks that forms a legal operator, this method decides its TokenType. More...

Protected Member Functions
UString	Text ()

UString	Text (int startPosition)

sealed override void	AfterNewline ()

override bool	SupportDotIndents ()

UString	GetUnescapedString (bool hasEscapes, bool isTripleQuoted)

UString	UnescapeSQStringValue (bool hasBackslash)

Symbol	ParseOp (out TokenType type)

Protected fields
TokenType	_type

NodeStyle	_style

int	_startPosition

UString	_textValue

Dictionary< Pair< UString, bool >, Symbol >	_typeMarkers = new Dictionary<Pair<UString, bool>, Symbol>()

Dictionary< UString, Pair < Symbol, TokenType > >	_opCache = new Dictionary<UString, Pair<Symbol, TokenType>>()

Member Function Documentation

static TokenType Loyc.Syntax.Les.Les3Lexer.GetOperatorTokenType ( UString op )

inlinestatic

Under the assumption that op is a sequence of punctuation marks that forms a legal operator, this method decides its TokenType.

override Maybe<Token> Loyc.Syntax.Les.Les3Lexer.NextToken ( )

inline

Scans the next token and returns information about it.

Returns: The next token, or null at the end of the source file.

Implements Loyc.Syntax.Lexing.ILexer< Token >.

References Loyc.Newline, Loyc.Syntax.Operator, and Loyc.Syntax.Lexing.Spaces.

static string Loyc.Syntax.Les.Les3Lexer.UnescapeQuotedString	(	ref UString	sourceText,
		Action< int, string >	onError,
		UString	indentation = `default(UString)`,
		bool	allowExtraIndent = `false`,
		bool	parseAsBytes = `true`
	)

inlinestatic

Parses a normal or triple-quoted string that still includes the quotes. Supports quote types '\'', '"' and '`'.

Parameters

sourceText	input text
onError	Called in case of parsing error (unknown escape sequence or missing end quotes)
indentation	Inside a triple-quoted string, any text following a newline is ignored as long as it matches this string. For example, if the text following a newline is "\t\t Foo" and this string is "\t\t\t", the tabs are ignored and " Foo" is kept.
allowExtraIndent	Enable EC#/LES triple-quoted string indent rules, which allow an additional one tab or three spaces of indent beyond what the identation parameter specifies.

Returns: The decoded string

This method recognizes LES and EC#-style string syntax. Firstly, it recognizes triple-quoted strings (''' """ ```). These strings enjoy special newline handling: the newline is always interpreted as
regardless of the actual kind of newline ( and
newlines come out as
), and indentation following the newline can be stripped out. Triple-quoted strings can have escape sequences that use both kinds of slash, like so: / / \'/ "/ \0/. However, there are no unicode escapes (/ is NOT supported).

Secondly, it recognizes normal strings (' " `). These strings stop parsing (with an error) at a newline, and can contain C-style escape sequences: \' " \0 etc. C#-style verbatim strings are NOT supported.

static void Loyc.Syntax.Les.Les3Lexer.UnescapeQuotedString	(	ref UString	sourceText,
		Action< int, string >	onError,
		StringBuilder	sb,
		UString	indentation = `default(UString)`,
		bool	allowExtraIndent = `false`,
		bool	parseAsBytes = `true`
	)

inlinestatic

Parses a normal or triple-quoted string that still includes the quotes (see documentation of the first overload) into a StringBuilder.

References Loyc.Localize.Localized().

static bool Loyc.Syntax.Les.Les3Lexer.UnescapeString	(	ref UString	sourceText,
		char	quoteType,
		bool	isTripleQuoted,
		Action< int, string >	onError,
		StringBuilder	sb,
		UString	indentation = `default(UString)`,
		bool	allowExtraIndent = `false`,
		bool	parseAsBytes = `true`
	)

inlinestatic

Parses a normal or triple-quoted string whose starting quotes have been stripped out. If triple-quote parsing was requested, stops parsing at three quote marks; otherwise, stops parsing at a single end-quote or newline.

Parameters

allowExtraIndent After each newline, indentation matching the previous line is ignored. When this parameter is true, an additional one tab or three spaces is ignored if the initial indent matched.

parseAsBytes If this is true, LES parsing mode is used. This mode assumes that the string represents a sequence of bytes, even though it is stored as WTF-16 in memory. In particular, it allows the escape sequence which represents a byte. If the byte NN is between 0x80 and 0xFF, it is encoded as 0xDC80 + NN. As a side effect, lone surrogate code units between 0xDC80 and 0xDCFF are recoded in UTF-8 form and then translated to three WTF-16 code units (each of which is between 0xDC80 and 0xDCFF). If this parameter is false, this behavior is turned off and is parsed the same way as .

Returns: true if parsing stopped at one or three quote marks, or false if parsing stopped at the end of the input string or at a newline (in a string that is not triple-quoted).

This method recognizes LES and EC#-style string syntax.

References Loyc.Syntax.PrintHelpers.EscapeCStyle().

Property Documentation

bool Loyc.Syntax.Les.Les3Lexer.PreferCustomLiterals

getset

If this flag is true, all literals except plain strings and true/false/null are stored as CustomLiteral, bypassing number parsing so that all original characters are preserved if the output is written back to text.

Properties

Public Member Functions

Static Public Member Functions

Protected Member Functions

Protected fields

Member Function Documentation

Property Documentation