Enhanced C#
Language of your choice: library documentation
Public fields | Public Member Functions | Static Public Member Functions | List of all members
Loyc.Syntax.Lexing.TokenTree Class Reference

A list of Token structures along with the ISourceFile object that represents the source file that the tokens came from. More...

Source file:
Inheritance diagram for Loyc.Syntax.Lexing.TokenTree:
Loyc.Collections.DList< Token > Loyc.Collections.IListSource< Token > Loyc.ICloneable< TokenTree >


A list of Token structures along with the ISourceFile object that represents the source file that the tokens came from.

This class is called TokenTree because certain kinds of tokens used by some parsers are formed into trees by using TokenTree as the type of the Token.Value of certain tokens. Specifically, the LES and EC# parsers expect open-bracket and open-brace tokens ('(', '[' and '{') to have a child TokenTree that contains all the tokens within a pair of brackets or braces. Typically this tree is not created directly by the lexer, but by a helper class (TokensToTree).

Caution: this class is mutable, even though TokenTrees are sometimes stored in LNodes which are supposed to be immutable. Please do not modify token trees that are stored inside LNodes.

Public fields

readonly ISourceFile File

Public Member Functions

 TokenTree (ISourceFile file, int capacity)
 TokenTree (ISourceFile file, ICollectionAndReadOnly< Token > items)
 TokenTree (ISourceFile file, IReadOnlyCollection< Token > items)
 TokenTree (ISourceFile file, ICollection< Token > items)
 TokenTree (ISourceFile file, IEnumerable< Token > items)
 TokenTree (ISourceFile file, Token[] items)
 TokenTree (ISourceFile file)
new TokenTree Clone ()
 Gets a deep (recursive) clone of the token tree. More...
TokenTree Clone (bool deep)
override string ToString ()
string ToString (Func< Token, ICharSource, string > toStringStrategy=null, ICharSource sourceCode=null)
override bool Equals (object obj)
bool Equals (TokenTree other)
 Compares the elements of the token tree for equality. More...
override int GetHashCode ()
LNodeList ToLNodes ()
 Converts this list of Token to a list of LNode. More...
DList< TokenFlatten ()
 Converts a token tree back to a plain list. More...
- Public Member Functions inherited from Loyc.Collections.DList< Token >
 DList (int capacity)
 DList (IReadOnlyCollection< T > items)
 DList (ICollection< T > items)
 DList (ICollectionAndReadOnly< T > items)
 DList (IEnumerable< T > items)
 DList (T[] items)
int IndexOf (T item)
void PushLast (ICollection< T > items)
void PushLast (IEnumerable< T > items)
void PushLast (IReadOnlyCollection< T > items)
void PushLast (ICollectionAndReadOnly< T > items)
void PushLast (T item)
void PushFirst (T item)
void PopLast (int amount)
void PopFirst (int amount)
void Insert (int index, T item)
void InsertRange (int index, ICollection< T > items)
void InsertRange (int index, IReadOnlyCollection< T > items)
void InsertRange (int index, ICollectionAndReadOnly< T > items)
void InsertRange (int index, IEnumerable< T > e)
void AddRange (ICollectionAndReadOnly< T > c)
void AddRange (ICollection< T > c)
void AddRange (IReadOnlyCollection< T > s)
void AddRange (IEnumerable< T > e)
void RemoveAt (int index)
void RemoveRange (int index, int amount)
bool TrySet (int index, T value)
TryGet (int index, out bool fail)
void Add (T item)
 An alias for PushLast(). More...
new int Add (object obj)
void Clear ()
bool Contains (T item)
void CopyTo (T[] array, int arrayIndex)
 Copies the elements of the collection to an Array, starting at a particular array index. More...
void CopyTo (int sourceIndex, T[] destination, int destinationIndex, int subcount)
void CopyTo (Array array, int arrayIndex)
bool Remove (T item)
new void Remove (object obj)
InternalDList< T >.Enumerator GetEnumerator ()
Maybe< T > TryPopFirst ()
Maybe< T > TryPeekFirst ()
Maybe< T > TryPopLast ()
Maybe< T > TryPeekLast ()
int BinarySearch (T k, Comparer< T > comp)
int BinarySearch< K > (K k, Func< T, K, int > comp)
void Resize (int newSize)
DList< T > Clone ()
DList< T > CopySection (int start, int subcount)
void Sort (Comparison< T > comp)
void Sort (int index, int count, Comparison< T > comp)
ListSlice< T > Slice (int start, int count)
- Public Member Functions inherited from Loyc.Collections.IListSource< Token >
IRange< T > Slice (int start, int count=int.MaxValue)
 Returns a sub-range of this list. More...
- Public Member Functions inherited from Loyc.ICloneable< TokenTree >
Clone ()

Static Public Member Functions

static LNode TokenToLNode (Token token, ISourceFile file)
 Converts a Token to a LNode. More...

Additional Inherited Members

- Properties inherited from Loyc.Collections.DList< Token >
int Capacity [get, set]
this[int index] [get, set]
this[int index, T defaultValue] [get]
int Count [get]
bool IsReadOnly [get]
First [get, set]
Last [get, set]
bool IsEmpty [get]
bool IsFixedSize [get]
bool IsSynchronized [get]
object SyncRoot [get]
- Protected fields inherited from Loyc.Collections.DList< Token >
InternalDList< T > _dlist

Member Function Documentation

◆ Clone()

new TokenTree Loyc.Syntax.Lexing.TokenTree.Clone ( )

Gets a deep (recursive) clone of the token tree.

◆ Equals()

bool Loyc.Syntax.Lexing.TokenTree.Equals ( TokenTree  other)

Compares the elements of the token tree for equality.

Because LNodes are compared by value and not by reference, and LNodes can contain TokenTrees, TokenTrees should also be compared by value.

◆ Flatten()

DList<Token> Loyc.Syntax.Lexing.TokenTree.Flatten ( )

Converts a token tree back to a plain list.

◆ TokenToLNode()

static LNode Loyc.Syntax.Lexing.TokenTree.TokenToLNode ( Token  token,
ISourceFile  file 

Converts a Token to a LNode.

fileThis becomes the LNode.Source property.

If you really need to store tokens as LNodes, use this. Only the Token.Kind, not the TypeInt, is preserved. Identifiers (where Kind==TokenKind.Id and Value is Symbol) are translated as Id nodes; everything else is translated as a call, using the TokenKind as the LNode.Name and the value, if any, as parameters. For example, if it has been treeified with TokensToTree, the token list for "Nodes".Substring(1, 3) as parsed by LES might translate to the LNode sequence String("Nodes"), Dot(@.), Substring, LParam(Number(1), Separator(@,), Number(3)), RParen(). The LNode.Range will match the range of the token.

References Loyc.Syntax.Lexing.Token.Children, Loyc.Syntax.Lexing.Token.EndIndex, Loyc.Syntax.Lexing.Token.Kind, Loyc.Syntax.Lexing.Token.Length, Loyc.Syntax.Lexing.Token.StartIndex, Loyc.Syntax.Lexing.Token.Style, and Loyc.Syntax.Lexing.Token.Value.

Referenced by Loyc.Syntax.Lexing.TokenTree.ToLNodes().

◆ ToLNodes()

LNodeList Loyc.Syntax.Lexing.TokenTree.ToLNodes ( )

Converts this list of Token to a list of LNode.

See Token.ToLNode(ISourceFile) for more information.

References Loyc.Syntax.Lexing.TokenTree.TokenToLNode().