Loyc.Syntax.Lexing.TokenTree Class Reference

A list of Token structures along with the ISourceFile object that represents the source file that the tokens came from. More...

---

Source file:

• /Core/Loyc.Syntax/Lexing/TokenTree.cs

Inheritance diagram for Loyc.Syntax.Lexing.TokenTree:

Remarks

A list of Token structures along with the ISourceFile object that represents the source file that the tokens came from.

This class is called TokenTree because certain kinds of tokens used by some parsers are formed into trees by using TokenTree as the type of the Token.Value of certain tokens. Specifically, the LES and EC# parsers expect open-bracket and open-brace tokens ('(', '[' and '{') to have a child TokenTree that contains all the tokens within a pair of brackets or braces. Typically this tree is not created directly by the lexer, but by a helper class (TokensToTree).

Caution: this class is mutable, even though TokenTrees are sometimes stored in LNodes which are supposed to be immutable. Please do not modify token trees that are stored inside LNodes.

Public fields
readonly ISourceFile	File

Public Member Functions
	TokenTree (ISourceFile file, int capacity)
	TokenTree (ISourceFile file, ICollectionAndReadOnly< Token > items)
	TokenTree (ISourceFile file, IReadOnlyCollection< Token > items)
	TokenTree (ISourceFile file, ICollection< Token > items)
	TokenTree (ISourceFile file, IEnumerable< Token > items)
	TokenTree (ISourceFile file, Token[] items)
	TokenTree (ISourceFile file)
new TokenTree	Clone ()
	Gets a deep (recursive) clone of the token tree. More...
TokenTree	Clone (bool deep)
override string	ToString ()
string	ToString (Func< Token, ICharSource, string >?toStringStrategy=null, ICharSource?sourceCode=null)
override bool	Equals (object?obj)
bool	Equals (TokenTree?other)
	Compares the elements of the token tree for equality. More...
override int	GetHashCode ()
LNodeList	ToLNodes ()
	Converts this list of Token to a list of LNode. More...
DList< Token >	Flatten ()
	Converts a token tree back to a plain list. More...

Static Public Member Functions
static LNode	TokenToLNode (Token token, ISourceFile file)
	Converts a Token to a LNode. More...

Member Function Documentation

new TokenTree Loyc.Syntax.Lexing.TokenTree.Clone

(

)

Gets a deep (recursive) clone of the token tree.

Implements Loyc.ICloneable< out T >.

bool Loyc.Syntax.Lexing.TokenTree.Equals ( TokenTree?  other )

inline

Compares the elements of the token tree for equality.

Because LNodes are compared by value and not by reference, and LNodes can contain TokenTrees, TokenTrees should also be compared by value.

DList<Token> Loyc.Syntax.Lexing.TokenTree.Flatten ( )

inline

Converts a token tree back to a plain list.

static LNode Loyc.Syntax.Lexing.TokenTree.TokenToLNode ( Token  token, ISourceFile  file  )

inlinestatic

Converts a Token to a LNode.

Parameters

file	This becomes the LNode.Source property.

If you really need to store tokens as LNodes, use this. Only the Token.Kind, not the TypeInt, is preserved. Identifiers (where Kind==TokenKind.Id and Value is Symbol) are translated as Id nodes; everything else is translated as a call, using the TokenKind as the LNode.Name and the value, if any, as parameters. For example, if it has been treeified with TokensToTree, the token list for "Nodes".Substring(1, 3) as parsed by LES might translate to the LNode sequence String("Nodes"), Dot(@.), Substring, LParam(Number(1), Separator(@,), Number(3)), RParen(). The LNode.Range will match the range of the token.

References Loyc.Syntax.Lexing.Token.EndIndex, Loyc.Syntax.Lexing.Token.Length, Loyc.Syntax.Lexing.Token.StartIndex, Loyc.Syntax.Lexing.Token.Style, and Loyc.Syntax.Lexing.Token.Value.

LNodeList Loyc.Syntax.Lexing.TokenTree.ToLNodes ( )

inline

Converts this list of Token to a list of LNode.

See Token.ToLNode(ISourceFile) for more information.