|
Enhanced C#
Loyc library documentation
|
|
Enhanced C#
Loyc library documentation
|
f Convenience methods for a fast pair of ISyncManager implementations intended for binary files/streams. The binary format is very fast and compact because it stores no metadata, but it must be used with care to avoid data corruption (see remarks). More...
f Convenience methods for a fast pair of ISyncManager implementations intended for binary files/streams. The binary format is very fast and compact because it stores no metadata, but it must be used with care to avoid data corruption (see remarks).
The binary ISyncManager implementations used by the static methods in this class do not support reordering or the NextField property, and they don't store metadata such as field names in the output. However, they do support deduplication (including cyclic object graphs).
Fields must be read/written in the same order they were read, and with the same data type (or a "compatible" data type as described below), or (at best) it will be impossible to read the data stream. At worst, the data stream could seem to be read successfully but produce a garbage result.
For this reason, you'll need to use versioning code (as explained below) if you can't ensure that the data will be read by the same version of your software that originally wrote it.
To understand the problem, suppose that in version 1 of your software, you write a class with an integer field called "SerialNumber":
So you write your synchronizer like this:
And you can serialize an example item like this:
The result is these 6 bytes: '(', 3, 0b11000001, 0b00000000, 0b00000011, ')'
The first byte is an object start marker (0x28). The second byte is the Id (0x03). The next three bytes are the SerialNumber (65539). The last byte is an end marker (0x29).
It's impossible to know just by looking at these bytes that they represent two numbers; in fact, by coincidence, if you serialize an object that contains the single floating- point number 3.7837386E-37f, the output is exactly the same 6 bytes! This is why you must read a binary data stream with the same schema you used to read it (same fields, same types, same order). If the schema is not exactly the same (and isn't compatible according to the rules listed below) then reading will fail in most cases, and in rare cases could produce a garbage answer.
Suppose that in version 2 of our software, we change the type of SerialNumber to string.
What happens if we naively read an old-format data stream after this type change? The outcome depends on the details of the old and new binary formats. If you use default settings, the string reader expects a list-start marker '[' followed by a length prefix. Since there is no '[' in the data stream, the reader throws an exception.
To avoid such an outcome, you should use a manual versioning process to make your synchronizer compatible with SyncBinary. The nice thing about manual versioning is that it not only maintains compatibility with other data formats such as JSON, but also adds clarification to your JSON output. So your code can be compatible with SyncBinary, SyncJson and protocol buffers all at the same time, while also allowing your data format to evolve over time.
One useful pattern is to write a format version number in a "top-level" object and then use that version number in child objects. For example, suppose that MyItem objects are always organized in groups:
Here's a properly versioned synchronizer for MyItemGroup and its child MyItems:
Here there is no version number in each individual MyItem object. Instead they inherit the version number from the Version property, which, in Reading mode, is normally read by the MyItemGroup Sync method. As usual, the MyItemGroup Sync method passes the MyItem Sync method as the third parameter when it calls sm.SyncList("Items", group.Items, Sync), so the value of Version is shared between them. Of course, if you deserialize MyItem directly using the MyItem Sync method, you will need to also set the Version property manually (or accept the default Version number).
The above code handles all versions of MyItem in a single function. If you prefer, you could have separate code paths for different versions:
This style of code is longer, but it does have an advantage: the old code for synchronizing V1 items has not been changed (except that MyItem is renamed to MyItemV1). The idea here is that it is easier to avoid accidentally changing the old schema if you avoid changing the old code.
The binary format is designed in such a way that certain data types/values are interchangeable. This means that in some cases it is possible to change certain data types into certain other data types while retaining backward compatibility (ability to read old versions of the format using new code that expects a different type than was originally stored).
short to int, or uint to ulong, or even int to BigInteger. (This rule does not apply to bitfields, which in general cannot change size.)bool to any integer type or nullable integer type.bool? to any nullable integer type.byte sbyte ushort short int uint to bool or bool?.byte? sbyte? ushort? short? int? uint? to bool?.char to ushort or vice versa.int to int?.float to float?, double to double?, or decimal to decimal?. Note: floating-point formats cannot be enlarged.All other type changes are unsafe and require you to write versioning code.
It's important to note, also, that changing the ObjectMode of a field is unsafe unless otherwise indicated. For example, adding or removing the ObjectMode.Deduplicate flag is unsafe unless the object, list or tuple being written begins with a marker (see Options.Markers).
Finally, you must not change certain settings of SyncBinary.Options between writing a data stream and reading it again. In particular, do not change Options.Markers, and don't change Options.RootMode in ways that affect the file format.
The following encoding schemes for key data types are "set in stone": they won't change between versions of SyncLib.
SyncBinary can optionally output markers at the start/end of an object. Their main purpose is simply to increase the chance that when a data stream is being read incorrectly (because you are not reading exactly the same fields/types that were written) an exception will occur soon afterward. Markers increase the data size, however.
All the markers are single ASCII bytes, e.g. if a marker is '{', that means 123. The start-object marker is '{' when entering an even-numbered Depth, or '(' if odd. The end-object marker is '}' when exiting an even-numbered Depth, or ')' otherwise. The start-list marker is always '['; the end-list marker is always ']'. Please note that strings count as lists of bytes, and will have list markers if they are enabled.
SyncBinary.Options.Markers controls which markers are read/written. Markers cannot be auto-detected (for example, if the data stream contains '{' there is no way to tell whether it's a marker or whether 0x7B is part of a data value). Therefore, when reading a binary data stream, you must use the same marker mode as when the blob was written, or reading will fail.
The default behavior is to write start/end markers for objects and type tags, and to write a start marker but no end marker for lists (including strings); the omitted end marker saves space.
A side effect of enabling start markers is that you can safely add or remove the ObjectMode.Deduplicate flag when writing an object that has a marker. By default, objects and lists have start markers but tuples do not.
If an object or list reference is null, it is written as a single 255 byte without any markers. The same thing is written when markers are disabled.
Both lists and normal objects can be written in tuple mode by including the ObjectMode.Tuple mode flag. This mode is useful for writing short fixed-length sequences of values in a compact way.
When writing a list in tuple mode, the list length is omitted from the output in order to make the output more compact. You must instead specify a tupleLength parameter when you call any of the SyncList extension methods.
Start/end markers can be enabled/disabled separately for tuples, and by default, markers are disabled. If you enable markers, the start/end markers are the same as for lists ('[' and ']'). If markers are disabled, the raw fields of the tuple are written to the output stream without any indication that the current object has changed.
If markers and deduplication are both disabled, it's possible to interchange a tuple of length N with N fields that have the same types. For example, if you write a tuple with two fields (an integer and a string), the data stream simply contains the integer followed by the string. Therefore, it is possible to change a normal integer and string field into a tuple without breaking backward compatibility (again, only if markers and deduplication are disabled). The same is true for ordinary objects, except that markers are enabled by default on ordinary objects.
When deduplication is enabled on a particular subobject or sublist, the object will have a deduplication marker right before the start marker (if any), unless the value written is null. The deduplication marker begins either with
The # or @ is followed by a variable-length integer which is an object ID. Object IDs are normally numbered starting from 1.
For example, suppose you write the same string reference twice using deduplication:
syncManager.Sync("1", "Hello", ObjectMode.Deduplicate);
syncManager.Sync("2", "Hello", ObjectMode.Deduplicate);
Given the default markers, the output bytes would be
'#', 0x01, '[', 0x05, 'H', 'e', 'l', 'l', 'o', '@', 0x01
Or in other words
35, 1, 91, 5, 72, 101, 108, 108, 111, 64, 1
If the start marker is enabled (as it is by default), then it is safe to add or remove the ObjectMode.Deduplicate flag between writing the data stream and reading it again. This is because SyncBinary.Reader can tell whether the Deduplicate mode was originally present when writing the stream: if it was, the first byte is '#' or '@', otherwise the first byte is '[' or 255 (meaning null). If the start marker is disabled, this doesn't work because (for example) it would be ambiguous whether '#' represents a deduplication marker or a list of length 35.
For strings in particular, you can change syncManager.Sync(fieldId, stringVal) to syncManager.Sync(fieldId, stringVal, ObjectMode.Deduplicate) safely if Markers.ListStart is enabled (as it is by default).
An object type tag is like a normal string field except that it is additionally prefixed with 'T' if the Markers.TypeTag option is enabled (as it is by default)
Integers are normally stored in big-endian format according to the variable- length scheme depicted below. It can be thought of as 9 separate formats: 7 of the formats are for integers of specific sizes, one is for very large integers, and one is for null. Each 'x' represents a number bit:
If the number is signed, the sign bit is always the first 'x' in the depiction. For example, the number -2 is represented as 0b01111110.
If there is a length prefix "n", it is encoded in the same way, so in principle there is no limit to the number of bits in the number. However, SyncBinary.Reader and SyncBinary.Writer limit the number size, by default, to 1 MB, large enough for (BigInteger)1 << (8 * 1024 * 1024). The length prefix is always 8 or more if produced by SyncLib. It can legally be smaller than 8, but not zero.
An example of the length-prefixed format is 254, 2, 1, 44. In this example, the length prefix is 2, and the number bits are 0x01 0x2C, which represents the number 300. However, the normal way of encoding 300 uses only two bytes (0x81, 0x2C).
This set of integer formats offers several advantages:
Int32 to Int64, without breaking backward compatibility. You can even upgrade fields to BigInteger! Note: although signed and unsigned numbers are stored in a similar way, switching an unsigned field to signed is unsafe. That's because certain positive numbers will be stored differently when signed vs unsigned. For example, signed 72 is a pair of bytes (0b10000000 0b01001000), but unsigned 72 is a single byte (0b01001000). If signed 72 is interpreted as unsigned, it's still 72, but if unsigned 72 is interpreted as signed, it becomes -56 because its most significant bit is a 1, which means "negative".
Integers in this format can be stored in a larger size than necessary (this can be called "non-canonical encoding"). For example, the number -3 can be stored in one byte as 0b01111101 or in two bytes as 0b10111111 0b11111101. SyncBinary.Writer writes integers in the shortest possible form (also known as the "canonical" form).
By the way, while variable-size integers use big-endian format, fixed-size numbers (e.g. floats) use little-endian format. There is a reason for this. Little-endian architectures such as x64 are the most common, and on these architectures, little-endian format is more natural and could lead to slightly higher performance. However, when writing variable-size integers, there are fewer than 8 bits in the top byte, so writing it in little-endian format would "scramble" it. For example, the number 0x12345 is written as hex C1 23 45, but in little-endian format the same number would be C5 1A 01 which is not only unrecognizable, but potentially could be slower to read/write depending on the implementation of the reader/writer.
The binary format also supports fixed-size little-endian bitfields via three special methods of ISyncManager: ISyncManager.Sync(FieldId, int, int, bool), ISyncManager.Sync(FieldId, long, int, bool) and ISyncManager.Sync(FieldId, BigInteger, int, bool).
If you always use a multiple of 8 as the number of bits, SyncBinary will store a whole number of bytes to represent each bitfield. For example, you can write 1234 as a 16-bit integer by calling Sync("ignored", 1234, 16, true). The first parameter is ignored. The boolean parameter is ignored when writing but causes sign-extension when reading. For example, if the input stream has a byte 0b1111_1111, Sync("...", 0, 8, true) would return it as -1 but Sync("...", 0, 8, false) would return it as 255.
If you use a fractional number of bytes then adjacent bitfields can share a single byte. For example, if you write a 20-bit bitfield followed by a 4-bit bitfield like so:
Then these three bytes are written: 0b00000011, 0b00000001, 0b11110000.
If the total number of bits in a series of bitfields is not a multiple of 8, then the high bits of the final byte are zero, and these high bits are ignored when the stream is read. For example, if your code says
Then these three bytes are written: 0b00000011, 0b00111101, 0b00000111. When reading these fields, the top two bits of the second byte are ignored.
Strings are stored in length-prefixed WTF-8 format (this means UTF-8 format, but highlights that any .NET string can be stored, even if it contains unpaired surrogate code units). The length is stored in the same format as integers (above). A length prefix of null represents the string being null. The length is expressed in bytes, not codepoints. This is essentially the same format as a byte array, so you could change the type of a string to a byte array while retaining backward compatibility.
If list markers are enabled, '[' is written before the string and ']' after. However, if the string is null, the markers are omitted (only 0xFF is written).
Surrogate pairs in the .NET representation are converted to single 4-byte characters in the UTF-8 format.
For example, suppose the string "Hello" is written, followed by a null reference, followed by a smiley "😀", using the default marker settings (i.e. list-start markers only). The bytes written will be
or in hexadecimal:
In .NET, the smiley "😀" is two UTF-16 code units (0xD83D 0xDE00), which is treated as a single code point U+1F600 that is converted to UTF-8 format.
If deduplication is enabled, the output will look approximately like this (object ID bytes may vary):
An individual char is saved the same way as a ushort (unsigned 16-bit integer). This is a variable-size format between 1 and 3 bytes. For example, the character 'A' is encoded as 0x41, and '•' (U+2022) is encoded as 0b1010_0000, 0b0010_0010.
A boolean is encoded as a signed integer where 0 = false and 1 = true (and as usual, a 255 byte means "null"). When reading a boolean from the data stream, it is read as a 32-bit signed integer, which is interpreted as false if it is 0 and true otherwise.
This means you can safely change an integer field to boolean or vice versa between versions. However, something can go wrong when reading an integer larger than 32 bits as a boolean. For example, if the integer is unsigned 0xFFFF_FFFF, it doesn't fit in a signed 32-bit integer. The reader's behavior then depends on the Options.ForReader.SilentlyTruncateLargeNumbers option. If it's true, it will be "truncated" to -1 and then converted to boolean true; otherwise, FormatException is thrown.
Floating point numbers are stored in 32-bit or 64-bit little-endian fixed-size IEEE 754 format.
Nullable floating point numbers are the same size, but use a specific NaN value to represent null: 0xFFF368E0 for 32-bit floats and 0xFFFE6C6C_756E06FE for 64-bit floats. 0xFFF368E0 typically shows up as "à hóÿ" in a hex editor ("ahoy", get it?), while 0xFFFE6C6C_756E06FE looks something like "þ nullþÿ". Why these particular values?
Since these NaNs have the high bit set, they are "quiet" NaNs on x64 and ARM.
Decimals are stored in the 128-bit fixed-size format defined by Microsoft for decimal numbers, but little-endian, so: the 96-bit "integer part" is stored first (it's analogous to the "mantissa" in floating-point, but is actually an integer), followed by 16 unused bits that are zero, followed by 8 bits for the negative base-10 exponent (which is limited to a range of 0 to 28) followed by the most significant byte, which is 0 if the number is positive and 0x80 if negative.
The most unique thing about this format is that zero bits at the end of a number are significant. For example, 7.00m is different from 7m (but equal to it). For 7m, the integer part is 7 and the negative exponent is 0; for 7.00m, the integer part is 700 and the negative exponent is 2.
The null value of decimal? is represented by 16 bytes that are all 0xFF. When reading a decimal?, the 13th and 14th bytes are checked to see if they are 0xFF, and if so, the value is treated as null. If they are neither 0xFF nor 0x00, an exception is thrown.
If list markers are enabled, '[' is written before a list and ']' afterward.
Arrays/lists are length-prefixed, and the length itself uses the standard integer format laid out above. For example, new[] { 1, 10, 100, 1000 } is encoded in seven bytes if markers are disabled (4, 1, 10, 0x80, 100, 0b10000011, 0b11101000) and eight bytes with a start marker ('[', 4, 1, 10, 0x80, 100, 0b10000011, 0b11101000). Null is encoded in a single byte (255). An empty array is also a single byte (0).
When writing null, list markers are omitted (only 0xFF is written).
Nested classes | |
| class | Options |
| Options that control general behavior of SyncBinary.Reader and SyncBinary.Writer. Note: some behaviors such as deduplication (including support for cyclic references) are controlled at the level of individual fields, and such options are not duplicated here. More... | |
| struct | Reader |
| An implementation of ISyncManager for reading SyncLib's compact, high-performance data format. Note that while SupportsDeduplication is true, SupportsReordering and SupportsNextField are false, the binary format is not self-describing, and field names/IDs are not stored in the data stream, so you must be careful to read the data in the same order and with compatible types as what was written (for details and guidelines, see SyncBinary). Please call SyncBinary.Read or SyncBinary.ReadI if you just want to serialize something, or NewWriter to create one of these. More... | |
| struct | Writer |
| The ISyncManager implementation for writing SyncLib's compact, high-performance binary data format (see SyncBinary for details and guidelines). Please call WriteI or Write if you just want to serialize something, or NewWriter to create one of these. More... | |
Public Types | |
| enum | FieldIdMode { FieldIdMode.Off = 0, FieldIdMode.Strings = 1, FieldIdMode.Integers = 2 } |
| The possible values of Options.FieldIdMode. More... | |
| enum | Markers { None = 0, ObjectStart = 1 << ObjectMode.Normal, ObjectEnd = 16 << ObjectMode.Normal, Objects = ObjectStart | ObjectEnd, ListStart = 1 << ObjectMode.List, ListEnd = 16 << ObjectMode.List, Lists = ListStart | ListEnd, TupleStart = 1 << ObjectMode.Tuple, TupleEnd = 16 << ObjectMode.Tuple, Tuples = TupleStart | TupleEnd, TypeTag = 256, Default = Objects | ListStart | TypeTag, All = Objects | Lists | Tuples | TypeTag } |
| Used to specify which marker bytes will be written or expected in a serialized SyncBinary data stream. More... | |
Static Public Member Functions | |
| static SyncBinary.Reader | NewReader (IScanner< byte > input, Options?options=null) |
| static SyncBinary.Reader | NewReader (ReadOnlyMemory< byte > input, Options?options=null) |
| static SyncBinary.Reader | NewReader (byte[] input, Options?options=null) |
| static T | Read< T > (ReadOnlyMemory< byte > input, SyncObjectFunc< Reader, T > sync, Options?options=null) |
| static T | ReadI< T > (ReadOnlyMemory< byte > input, SyncObjectFunc< ISyncManager, T > sync, Options?options=null) |
| static T | Read< T, SyncObject > (ReadOnlyMemory< byte > input, SyncObject sync, Options?options=null) |
| static T | Read< T > (byte[] input, SyncObjectFunc< Reader, T > sync, Options?options=null) |
| static T | ReadI< T > (byte[] input, SyncObjectFunc< ISyncManager, T > sync, Options?options=null) |
| static T | Read< T, SyncObject > (byte[] input, SyncObject sync, Options?options=null) |
| static ReadOnlyMemory< byte > | Write< T > (T value, SyncObjectFunc< Writer, T > sync, Options?options=null) |
| static ReadOnlyMemory< byte > | WriteI< T > (T value, SyncObjectFunc< ISyncManager, T > sync, Options?options=null) |
| static ReadOnlyMemory< byte > | Write< T, SyncObject > (T value, SyncObject sync, Options?options=null) |
| static SyncBinary.Writer | NewWriter (IBufferWriter< byte >?output=null, Options?options=null) |
The possible values of Options.FieldIdMode.
| Enumerator | |
|---|---|
| Off |
Field identifiers are not read or written. This is the most dangerous mode, but it's also the simplest, the most efficient and the most compact. Plus, this mode is needed for reading most 3rd-party and legacy file formats. In this mode, the field ID parameter is ignored in all calls to the Sync() methods, and only data values are written to the file. For example, suppose The default output will be 7 bytes: 5, 'L', 'i', 'm', 'e', 123, 1. As you can see, the file format is "raw", with no delimiters or schema to help indicate the meaning of the bytes. This works perfectly fine as long as the same code is used to read the stream that was used to write the stream. You can even support versioning pretty easily by explicitly storing a version number in the stream. For example, suppose that the "Number" field was added in version 2. You can support both versions 1 and 2 using code like this: public class ExampleObject { public int Number; public string? Fruit; public bool Happy; } public class ExampleSync<SM> : ISyncObject<SM, ExampleObject> where SM: ISyncManager { public int Version = 2; public ExampleObject Sync(SM sm, ExampleObject? value) { value ??= new ExampleObject(); Version = sm.Sync("Version", Version); if (Version > 2 || Version < 1) throw new FormatException("Unrecognized data format"); sm.SyncRef("Fruit", ref value.Fruit); if (Version >= 2) sm.SyncRef("Number", ref value.Number); sm.SyncRef("Happy", ref value.Happy); return value; } } |
| Strings |
Field identifiers are read and written as strings. A side effect is that ISyncManager.SupportsNextField will return true, although ISyncManager.SupportsReordering remains false. As usual, list/tuple items don't have a field ID even in this mode. When reading a stream, an exception is thrown if the field ID doesn't match the string in the FieldId, except if the FieldId is null, in which case the field ID string is read but ignored. If you write a field with a null field ID, that null value is still written to the file, not suppressed, so that ISyncManager.NextField still works properly when reading the file. |
| Integers |
Field identifiers are read and written as integers. A side effect is that ISyncManager.SupportsNextField and ISyncManager.NeedsIntegerIds will both return true, although ISyncManager.SupportsReordering will remain false. As usual, list/tuple items don't have a field ID even in this mode. When reading a stream, an exception is thrown if the field ID doesn't match the integer in the FieldId, except if the FieldId lacks an integer, in which case the field ID number is read but ignored. If no integer FieldId is provided when writing the file, the number N+1 is written instead, where N is zero if this is the first field in an object, or N is the last field number to have been written in the same object. |
| enum SyncBinary.Markers |
Used to specify which marker bytes will be written or expected in a serialized SyncBinary data stream.
1.8.7