5. A brief overview of the Loyc libraries

30 May 2016

When writing a parser, you have to decide whether you’ll use the Loyc runtime libraries or not; the main advantage of not using them is that you won’t have to distribute the 3 Loyc DLLs with your application. But they contain a lot of useful stuff, so have a look and see if you like them.

The alternative is to use the standalone LexerSource and ParserSource which can be found in the “standalone” example in the LLLPG Samples repository.

The important library for parsers based on LLLPG is Loyc.Syntax.dll, which depends on Loyc.Essentials.dll and Loyc.Collections.dll. These DLLs have documentation for most of the classes they contain, automatically available to VS IntelliSense through Loyc.Syntax.xml, Loyc.Essentials.xml and Loyc.Collections.xml; you can also view the reference documentation online. The Loyc libraries contain only “safe”, verifiable code.

In brief, let me just say very briefly what these libraries are for and what they contain.

Loyc.Essentials.dll

A library of general-purpose code that supplements the .NET BCL (standard libraries). It contains the following categories of stuff:

• Collection stuff: interfaces, adapters, helper classes, base classes, extension methods, and implementations for simple “core” collections such as InternalList. You can learn more in the docs, but note that the documentation also shows the collections from Loyc.Collections.dll since it’s all in the same namespace, Loyc.Collections.
• Other utilities: message sinks (IMessageSink), Symbol, threading stuff, a miniture clone of NUnit (MiniTest, RunTests), and miscellaneous [“global” functions] and extension methods. Compatibility: a very small amount of .NET 4.5 stuff, backported to .NET 4.0 when using the .NET 4 build.

Loyc.Essentials also defines ICharSource, a standard interface for a source of characters, which is used by lexers. It also defines UString which is a string slice structure that implements ICharSource. string is implicitly convertible to UString. The Slice(start, count) extension method for string can also get slices.

IMessageSink serves as a simple, generic logging interface. It is recommended that your parsers report warnings and errors to an IMessageSink object. You can use ConsoleMessageSink.Value to print (colored) errors to the console, MessageSink.Null to suppress output, and MessageSink.FromDelegate((type, context, message, args) => {...}) to customize error handling.

The ParseHelpers class has generic number parsers that are handy for lexers, such as TryParseDouble, which can parse numbers of any reasonable radix and is therefore useful for hex float literals such as 0xF.Fp+1 (a syntax that represents 31.875).

Loyc.Collections.dll

A library of data structures, mostly rather complex ones, currently all written by me:

• VLists: this data structure is notable because Loyc nodes (LNodes) use VList<LNode> for their arguments and attributes. This is an implementation detail that ideally you wouldn’t have to know about; but C# has no typedefs that I could use to hide the type, and since VLists are structs, if you treat them as IList<T> they will be boxed, and you don’t really want that.
• ALists, including the B+tree-like data structures BList<T>, BDictionary<K,V>, and my favorite, BMultiMap<K,V>, plus the new SparseAList<T> which I use in my syntax highlighter.
• Bijection<K1,K2>: A dictionary that goes in both directions.
• And more!

Loyc.Syntax.dll

Provides the foundations for LLLPG and contains the reference implementation of LES, the syntax tree interchange format:

• BaseLexer is the recommended base class for lexers created with LLLPG. BaseParserForList<Token,MatchType> is the recommended base class for parsers.
• StreamCharSource is an implementation of ICharSource designed for parsing a file without storing the whole thing in memory.
• ISourceFile encapsulates an ICharSource, a file name string, and a mapping to translate character indexes to (line, column) pairs and back. It is derived from IIndexPositionMapper.
• SourceRange is a triple (ISourceFile Source, int StartIndex, int Length) that represents a range of characters in a source file.
• SourcePos is a (filename, line, column) triple. While SourceRange is a struct so it can be stored compactly, SourcePos is assumed to be used much less often, and it is a class so it can be derived from LineAndCol which is a (line, column) pair.
• IndexPositionMapper provides mapping from SourceRange to SourcePos and back, but you don’t necessarily need this class because BaseLexer already keeps track of the current line number (and where it started). In your lexer, you must call AfterNewline() at each newline in order for index-position mapping to work correctly.
• LNode is a Loyc Tree. Parsers commonly use LNodeFactory to help construct LNodes.
• LesLanguageService.Value provides an LES parser and printer. It implements IParsingService.
• SourceFileWithLineRemaps is a helper class for languages that have a #line directive.
• Precedence: a simple but flexible standard representation for the concept of operator precedence and “miscibility”.
• CodeSymbols is a static class filled with standard Symbols used in Loyc trees for operators (Add for +, Sub for -, Mul for *, Assign for =, Eq for ==, …), statements (Class for #class, Enum for #enum, ForEach for #foreach, …), modifiers (Private for #private, Static for #static, Virtual for #virtual, …), types (Void for #void, Int32 for #int32, Double for #double, …), and so on.

Next up

The next article in this series is “how to write a parser”.