A list of Token structures along with the ISourceFile object that represents the source file that the tokens came from. More...

Source file:

/Core/Loyc.Syntax/Lexing/TokenTree.cs

Inheritance diagram for Loyc.Syntax.Lexing.TokenTree:

Remarks

A list of Token structures along with the ISourceFile object that represents the source file that the tokens came from.

This class is called TokenTree because certain kinds of tokens used by some parsers are formed into trees by using TokenTree as the type of the Token.Value of certain tokens. Specifically, the LES and EC# parsers expect open-bracket and open-brace tokens ('(', '[' and '{') to have a child TokenTree that contains all the tokens within a pair of brackets or braces. Typically this tree is not created directly by the lexer, but by a helper class (TokensToTree).

Caution: this class is mutable, even though TokenTrees are sometimes stored in LNodes which are supposed to be immutable. Please do not modify token trees that are stored inside LNodes.

Public fields
readonly ISourceFile	File

Public Member Functions
	TokenTree (ISourceFile file, int capacity)

	TokenTree (ISourceFile file, ICollectionAndReadOnly< Token > items)

	TokenTree (ISourceFile file, IReadOnlyCollection< Token > items)

	TokenTree (ISourceFile file, ICollection< Token > items)

	TokenTree (ISourceFile file, IEnumerable< Token > items)

	TokenTree (ISourceFile file, Token[] items)

	TokenTree (ISourceFile file)

new TokenTree	Clone ()
	Gets a deep (recursive) clone of the token tree. More...

TokenTree	Clone (bool deep)

override string	ToString ()

string	ToString (Func< Token, ICharSource, string > toStringStrategy=null, ICharSource sourceCode=null)

override bool	Equals (object obj)

bool	Equals (TokenTree other)
	Compares the elements of the token tree for equality. More...

override int	GetHashCode ()

LNodeList	ToLNodes ()
	Converts this list of Token to a list of LNode. More...

DList< Token >	Flatten ()
	Converts a token tree back to a plain list. More...

Public Member Functions inherited from Loyc.Collections.DList< Token >
	DList (int capacity)

	DList (IReadOnlyCollection< T > items)

	DList (ICollection< T > items)

	DList (ICollectionAndReadOnly< T > items)

	DList (IEnumerable< T > items)

	DList (T[] items)

int	IndexOf (T item)

void	PushLast (ICollection< T > items)

void	PushLast (IEnumerable< T > items)

void	PushLast (IReadOnlyCollection< T > items)

void	PushLast (ICollectionAndReadOnly< T > items)

void	PushLast (T item)

void	PushFirst (T item)

void	PopLast (int amount)

void	PopFirst (int amount)

void	Insert (int index, T item)

void	InsertRange (int index, ICollection< T > items)

void	InsertRange (int index, IReadOnlyCollection< T > items)

void	InsertRange (int index, ICollectionAndReadOnly< T > items)

void	InsertRange (int index, IEnumerable< T > e)

void	AddRange (ICollectionAndReadOnly< T > c)

void	AddRange (ICollection< T > c)

void	AddRange (IReadOnlyCollection< T > s)

void	AddRange (IEnumerable< T > e)

void	RemoveAt (int index)

void	RemoveRange (int index, int amount)

bool	TrySet (int index, T value)

T	TryGet (int index, out bool fail)

void	Add (T item)
	An alias for PushLast(). More...

new int	Add (object obj)

void	Clear ()

bool	Contains (T item)

void	CopyTo (T[] array, int arrayIndex)
	Copies the elements of the collection to an Array, starting at a particular array index. More...

void	CopyTo (int sourceIndex, T[] destination, int destinationIndex, int subcount)

void	CopyTo (Array array, int arrayIndex)

bool	Remove (T item)

new void	Remove (object obj)

InternalDList< T >.Enumerator	GetEnumerator ()

Maybe< T >	TryPopFirst ()

Maybe< T >	TryPeekFirst ()

Maybe< T >	TryPopLast ()

Maybe< T >	TryPeekLast ()

int	BinarySearch (T k, Comparer< T > comp)

int	BinarySearch< K > (K k, Func< T, K, int > comp)

void	Resize (int newSize)

DList< T >	Clone ()

DList< T >	CopySection (int start, int subcount)

void	Sort (Comparison< T > comp)

void	Sort (int index, int count, Comparison< T > comp)

ListSlice< T >	Slice (int start, int count)

Public Member Functions inherited from Loyc.Collections.IListSource< Token >
IRange< T >	Slice (int start, int count=int.MaxValue)
	Returns a sub-range of this list. More...

Public Member Functions inherited from Loyc.ICloneable< TokenTree >
T	Clone ()

Static Public Member Functions
static LNode	TokenToLNode (Token token, ISourceFile file)
	Converts a Token to a LNode. More...

Additional Inherited Members
Properties inherited from Loyc.Collections.DList< Token >
int	Capacity `[get, set]`

T	this[int index] `[get, set]`

T	this[int index, T defaultValue] `[get]`

int	Count `[get]`

bool	IsReadOnly `[get]`

T	First `[get, set]`

T	Last `[get, set]`

bool	IsEmpty `[get]`

bool	IsFixedSize `[get]`

bool	IsSynchronized `[get]`

object	SyncRoot `[get]`

Protected fields inherited from Loyc.Collections.DList< Token >
InternalDList< T >	_dlist

Member Function Documentation

◆ Clone()

new TokenTree Loyc.Syntax.Lexing.TokenTree.Clone ( )

Gets a deep (recursive) clone of the token tree.

◆ Equals()

bool Loyc.Syntax.Lexing.TokenTree.Equals ( TokenTree other )

inline

Compares the elements of the token tree for equality.

Because LNodes are compared by value and not by reference, and LNodes can contain TokenTrees, TokenTrees should also be compared by value.

◆ Flatten()

DList<Token> Loyc.Syntax.Lexing.TokenTree.Flatten ( )

inline

Converts a token tree back to a plain list.

◆ TokenToLNode()

static LNode Loyc.Syntax.Lexing.TokenTree.TokenToLNode	(	Token	token,
		ISourceFile	file
	)

inlinestatic

Converts a Token to a LNode.

Parameters

file	This becomes the LNode.Source property.

If you really need to store tokens as LNodes, use this. Only the Token.Kind, not the TypeInt, is preserved. Identifiers (where Kind==TokenKind.Id and Value is Symbol) are translated as Id nodes; everything else is translated as a call, using the TokenKind as the LNode.Name and the value, if any, as parameters. For example, if it has been treeified with TokensToTree, the token list for "Nodes".Substring(1, 3) as parsed by LES might translate to the LNode sequence String("Nodes"), Dot(@.), Substring, LParam(Number(1), Separator(@,), Number(3)), RParen(). The LNode.Range will match the range of the token.

References Loyc.Syntax.Lexing.Token.Children, Loyc.Syntax.Lexing.Token.EndIndex, Loyc.Syntax.Lexing.Token.Kind, Loyc.Syntax.Lexing.Token.Length, Loyc.Syntax.Lexing.Token.StartIndex, Loyc.Syntax.Lexing.Token.Style, and Loyc.Syntax.Lexing.Token.Value.

Referenced by Loyc.Syntax.Lexing.TokenTree.ToLNodes().