Encapsulates an algorithm that consumes trivia (comments and newlines) from a list and adds it as trivia attributes into LNodes. More...

Source file:

/Core/Loyc.Syntax/Parsing/AbstractTriviaInjector.cs

Remarks

Encapsulates an algorithm that consumes trivia (comments and newlines) from a list and adds it as trivia attributes into LNodes.

Call Run to invoke the algorithm. One must also write a derived class that knows how to interpret the trivia and associate it with a specific LNode, or use the standard derived class StandardTriviaInjector.

The algorithm is designed to postprocess output from a parser that works in a typical way. The lexer for your language needs to follow the following rules:

It must not include the newline character in the range of a single-line comment.
Generally, newlines (including the newline after a single-line comment) should be included in the trivia list, as the algorithm relies on newline trivia to notice when a new line is starting. Notably, a comment/trivia right after a statement (e.g. on the same line) should normally be associated with that statement; if this class is unaware of the newline it will be associated with the next statement instead.

Typically one will wrap the lexer in Lexing.TriviaSaver, which saves trivia while filtering out whitespace so that the parser doesen't see it.

Your language's parser needs to follow the following rules:

The parser should assign minimal boundaries to each node: the LNode.Range should not be wider than necessary. If there is a comment before an expression like /* ! * / x + y, the parser should not include the comment as part of the range unless it wants the comment to be associated with a child node (x) instead of with the entire expression (x + y).
However, if a node has normal (non-trivia) attributes attached to it, the Range of the node must include those attributes. If your parser fails to do this, one symptom can be that a newline after an attribute "moves up" in the attribute list so it appears before the attribute.
The parser should use LNodeFactory.InParens to place a node in parentheses and include the index of the opening and closing parens.
If an expression/statement is terminated by a semicolon/comma, it's best (but not crucial) to include the semicolon/comma in the LNode.Range, but if a statement is terminated by a newline, the newline should not be included in the range. If general, when the terminator is included in the range, then any comment that appears before the terminator is attached to the final child node rather than to the node as a whole. (e.g. in x = y /*y* /;, the comment is associated with y if the comment is within the range of the statement as a whole.)

This example shows how comments are associated with nodes: [NOTE: the space in "* /" is a workaround for a serious bug in Doxygen, the html doc generator]

// Comment attached to block
{
  // Comment attached to Foo() call
  Foo(
    // Comment attached to «argument1 + 1». Note that the comma is 
    // generally invisible to the injector since it is not part of the LNode 
    // tree, so the 1st and 2nd comments will both be attached to the sum-
    // expression as a whole.
    argument1 + 1 /*1st comment* /, // 2nd comment
    // Comment attached to «argument2 + 2»
    argument2 + 2); // Comment attached to Foo() call
  Area = 3.14159265/*PI* /  *  /*radius* /r**2;
  // Comment attached to «Area =» statement, preceded by newline trivia?

  // Comment attached to Bar()
  @BarAttr // Comment attached to BarAttr attribute
  Bar();

  // Comment attached to Bar() because there are no statements afterward
}

When there is a single newline between two nodes, Run will associate it with the second one. When there is a blank line between two nodes (two newlines), the first newline (and any before) is associated with the first node and the second newline (and any following) is associated with the second.

Properties
IListSource< Trivia >	SortedTrivia `[get, set]`
	List of trivia to be injected by Run. Must be sorted. More...

int	NextIndex `[get, set]`
	Index of next trivia to be injected. More...

Public Member Functions
	AbstractTriviaInjector (IListSource< Trivia > sortedTrivia)
	Initializes the SortedTrivia property. More...

IEnumerator< LNode >	Run (IEnumerator< LNode > nodes)
	Attaches trivia to the input nodes provided. More...

Protected Member Functions
abstract LNodeList	GetTriviaToAttach (LNode node, IListSource< Trivia > trivia, TriviaLocation loc, LNode parent, int indexInParent)
	Derived class translates a list of trivia (tokens) into appropriate trivia attributes. This will be called for leading trivia before trailing trivia. More...

virtual void	ProcessChildrenOfOneLiner (ref LNode node)
	This method is called when a node has no newlines or comments within it (although the node may still have a leading or trailing comment). The method should add appendStatement trivia inside blocks in the node, if necessary. More...

abstract SourceRange	GetRange (Trivia trivia)
	Gets the SourceRange for an element of trivia. More...

abstract bool	IsNewline (Trivia trivia)
	Returns true if the trivia represents a newline, false otherwise. More...

virtual LNode	GetEmptyResultSet ()
	A method called to create a virtual node, in order to apply trivia to a source file that is completely empty except for trivia. More...

virtual LNode	DoneAttaching (LNode node, LNode parent, int indexInParent)
	This method is called after a node has been processed and any applicable trivia was attached. More...

IEnumerator< Pair< LNode, int > >	RunCore (IEnumerator< Pair< LNode, int >> nodes, LNode parent)
	Core trivia associaton algorithm. More...

Static Protected Member Functions
static IEnumerator< Pair< T, int > >	WithIndexes< T > (IEnumerator< T > e)

Constructor & Destructor Documentation

◆ AbstractTriviaInjector()

Loyc.Syntax.AbstractTriviaInjector< Trivia >.AbstractTriviaInjector ( IListSource< Trivia > sortedTrivia )

inline

Initializes the SortedTrivia property.

Member Function Documentation

◆ DoneAttaching()

virtual LNode Loyc.Syntax.AbstractTriviaInjector< Trivia >.DoneAttaching	(	LNode	node,
		LNode	parent,
		int	indexInParent
	)

inlineprotectedvirtual

This method is called after a node has been processed and any applicable trivia was attached.

Parameters

node	Node (after trivia attached)
parent	Parent of `node` (old version, before changes to children are applied)
indexInParent	Index of `node` within `parent`.

Returns: Should return node or an altered version of node.

This method gives the derived class one final chance to rearrange or alter the interpretation of the attached trivia. Note that this method may be called on some nodes to which trivia was not attached, when siblings of the same parent had trivia attached.

Reimplemented in Loyc.Ecs.Parser.EcsTriviaInjector.

◆ GetEmptyResultSet()

virtual LNode Loyc.Syntax.AbstractTriviaInjector< Trivia >.GetEmptyResultSet ( )

inlineprotectedvirtual

A method called to create a virtual node, in order to apply trivia to a source file that is completely empty except for trivia.

Default implementation attaches all trivia to a "missing" node (zero-length identifier). If this method returns null, the source file will truly be empty (containing no trivia either).

Referenced by Loyc.Syntax.AbstractTriviaInjector< Token >.Run().

◆ GetRange()

abstract SourceRange Loyc.Syntax.AbstractTriviaInjector< Trivia >.GetRange ( Trivia trivia )

protectedpure virtual

Gets the SourceRange for an element of trivia.

Referenced by Loyc.Syntax.AbstractTriviaInjector< Token >.GetEmptyResultSet().

◆ GetTriviaToAttach()

abstract LNodeList Loyc.Syntax.AbstractTriviaInjector< Trivia >.GetTriviaToAttach	(	LNode	node,
		IListSource< Trivia >	trivia,
		TriviaLocation	loc,
		LNode	parent,
		int	indexInParent
	)

protectedpure virtual

Derived class translates a list of trivia (tokens) into appropriate trivia attributes. This will be called for leading trivia before trailing trivia.

Parameters

node	The node.
trivia	Trivia to be associated with `node`.
loc	Location of the trivia. For a given node, the base class calls this method at most once for each value of TriviaLocation.
parent	(Original version of) parent of `node`.
indexInParent	Index of `node` within `parent`.

Returns: A list of trivia attributes that should be attached to the node.

This method will be called for a node when there is no trivia associated with that node; the derived class may, for example, want to add appendStatement in certain cases.

Referenced by Loyc.Syntax.AbstractTriviaInjector< Token >.RunCore().

◆ IsNewline()

abstract bool Loyc.Syntax.AbstractTriviaInjector< Trivia >.IsNewline ( Trivia trivia )

protectedpure virtual

Returns true if the trivia represents a newline, false otherwise.

Referenced by Loyc.Syntax.AbstractTriviaInjector< Token >.RunCore().

◆ ProcessChildrenOfOneLiner()

virtual void Loyc.Syntax.AbstractTriviaInjector< Trivia >.ProcessChildrenOfOneLiner ( ref LNode node )

inlineprotectedvirtual

This method is called when a node has no newlines or comments within it (although the node may still have a leading or trailing comment). The method should add appendStatement trivia inside blocks in the node, if necessary.

Referenced by Loyc.Syntax.AbstractTriviaInjector< Token >.ProcessChildrenOfOneLiner(), and Loyc.Syntax.AbstractTriviaInjector< Token >.RunCore().

◆ Run()

IEnumerator<LNode> Loyc.Syntax.AbstractTriviaInjector< Trivia >.Run ( IEnumerator< LNode > nodes )

inline

Attaches trivia to the input nodes provided.

Parameters

nodes List of input nodes. This method calls nodes.MoveNext() before calling nodes.Current.

Trailing trivia after all nodes is attached to the final node. If nodes is empty then GetEmptyResultSet is called to create a dummy node and attach all trivia to it.

◆ RunCore()

IEnumerator<Pair<LNode, int> > Loyc.Syntax.AbstractTriviaInjector< Trivia >.RunCore	(	IEnumerator< Pair< LNode, int >>	nodes,
		LNode	parent
	)

inlineprotected

Core trivia associaton algorithm.

NOTE: the enumerator may DRIVE lexing and actually cause the trivia list (SortedTrivia) to increase in size. For this reason, this algorithm is careful to call nodes.MoveNext() BEFORE getting the current trivia. I'm not sure if this precaution is sufficient to preserve trivia in all "streaming" cases, but it has worked fine up to now.

Referenced by Loyc.Syntax.AbstractTriviaInjector< Token >.Run().