Encodes and decodes BAIS (Byte Array In String) encoding, which preserves runs of ASCII characters unchanged. This encoding is useful for debugging (since ASCII runs are visible) and for conversion of bytes to JSON. More...

Source file:

/Core/Loyc.Essentials/Utilities/ByteArrayInString.cs

Remarks

Encodes and decodes BAIS (Byte Array In String) encoding, which preserves runs of ASCII characters unchanged. This encoding is useful for debugging (since ASCII runs are visible) and for conversion of bytes to JSON.

Arrays encoded with ByteArrayInString.Convert(ArraySlice{byte}, bool) tend to be slightly more compact than standard Uuencoding or Base64, and when you use this encoding in JSON with UTF-8, the output is typically also more compact than yEnc since double-byte characters above 127 are avoided.

A BAIS string alternates between runs of "direct" bytes (usually bytes in the ASCII range that are represented as themselves) and runs of a special base-64 encoding. The base-64 encoding is a sequence of 6-bit digits with 64 added to them, except for 63 which is mapped to itself. This is easier and faster to encode and decode than standard Base64 and has an interesting property described below.

A BAIS string begins in ASCII mode and switches to base 64 when the '' character is encountered. Base-64 mode ends, returning to ASCII, when a '!' character is encountered.

For example:

  //                    C   a    t       

  E        A   B   C   D
  var b = new byte[] { 67, 97, 116, 131, 10, 69, 255, 65, 66, 67, 68 };
  Assert.AreEqual(ByteArrayInString.Convert(b), "Cat\b`piE?tEB!CD");

A byte sequence such as 131, 10, 69, 255 can be encoded in base 64 as illustrated:

             ---131---    ---10----    ---69----  ---255---  
  Bytes:     1000 0011    0000 1010    0100 0101  1111 1111  
  Base 64:   100000   110000   101001    000101   111111   110000
  Encoded: 01100000 01110000 01101001  01000101 01111111 01110000
           ---96--- --112--- --105---  ---69--- --127--- --112---
              `        p        i         E        ~        p

An interesting property of this base-64 encoding is that when it encodes bytes between 63 and 126, those bytes appear unchanged at certain offsets (specifically the third, sixth, ninth, etc.) In this example, since the third byte is 'E' (69), it also appears as 'E' in the output.

When viewing BAIS strings, another thing to keep in mind is that runs of zeroes ('\0') will tend to appear as runs of @ characters in the base 64 encoding, although a single zero is not always enough to make a @ appear. Runs of 255 will tend to appear as runs of ?.

Since a BAIS string starts in ASCII mode by default, a sequence of bytes is normally represented as itself if it happens to be ASCII, e.g. the byte form of "Hello!" is encoded as the string "Hello!". However, in order to unambiguously distinguish BAIS from conventional base64 encoding, the first character of any BAIS string can be set to '!'. For example, "!Hello!" is also a valid BAIS encoding of "Hello!". Consequently, if the first byte encoded is '!' (33), the BAIS encoding will start with two '!' characters.

There are many ways to encode a given byte array as BAIS. It is possible to guarantee that a BAIS encoding is never more than one character larger than conventional base64, but the easiest way to guarantee this is to avoid switching to ASCII mode unless there are at least 6 ASCII characters in a row. This implementation requires only 4 ASCII characters in a row instead, so in pathological cases the result can be 7% longer than conventional base64, but this should be very rare. When the string is encoded to JSON, this pathological worst case is 15% longer than base64.

Static Public Member Functions
static string	ConvertFromBytes (ReadOnlySpan< byte > span, bool allowControlChars, bool forceInitialEscape=false)
	Encodes a byte array to a string with BAIS encoding, which preserves runs of ASCII characters unchanged. More...

static string	ConvertFromBytes (byte[] bytes, bool allowControlChars, bool forceInitialEscape=false)

static ArraySlice< byte >	ConvertToBytes (string s)
	Decodes a BAIS string back to a byte array. More...

static ArraySlice< byte >	ConvertToBytes (UString s)
	Decodes a BAIS string back to a byte array. More...

static ArraySlice< byte >	TryConvertToBytes (UString s)
	Decodes a BAIS string back to a byte array. More...

static ArraySlice< byte >	TryConvertToBytes (string s)
	Decodes a BAIS string back to a byte array. More...

static ArraySlice< byte >	TryConvertToBytes (ReadOnlySpan< byte > ascii)

static ArraySlice< byte >	TryConvertToBytesInPlace (Memory< byte > ascii)

static char	EncodeBase64Digit (int digit)

static int	DecodeBase64Digit (char digit)

static string	Convert (ArraySlice< byte > bytes, bool allowControlChars=true)

static ArraySlice< byte >	TryConvert (UString s)

static ArraySlice< byte >	TryConvert (string s)

static ArraySlice< byte >	Convert (string s)

static ArraySlice< byte >	Convert (UString s)

Member Function Documentation

static string Loyc.ByteArrayInString.ConvertFromBytes	(	ReadOnlySpan< byte >	span,
		bool	allowControlChars,
		bool	forceInitialEscape = `false`
	)

inlinestatic

Encodes a byte array to a string with BAIS encoding, which preserves runs of ASCII characters unchanged.

Parameters

allowControlChars	If true, control characters under 32 are treated as ASCII (except character 8 ''). When preparing output for JSON it may be more efficient to use false here, because JSON does not officially allow control characters in strings, so if this is true, a standard JSON encoder will lengthen control characters to 6-character escape sequences.
forceInitialEscape	If true, the first output character will always be '!' or ' ' to indicate that BAIS encoding is being used rather than conventional base64 encoding.

Returns: The encoded string.

If the byte array can be interpreted as ASCII, it is returned as characters, e.g. Convert(new byte[] { 65,66,67,33 }) == "ABC!". When non-ASCII bytes are encountered, they are encoded as described in the description of this class.

For simplicity, this method's base-64 encoding always encodes groups of three bytes if possible (as four characters). This decision may, unfortunately, cut off the beginning of some ASCII runs. Also, to ensure that the encoding is never significantly larger than base64, a switch to ASCII mode only happens if there are at least 4 ASCII characters in a row. Because of these two facts combined, there are cases in which a string of 5 ASCII characters will be encoded as base64. But if there are at least 6 ASCII characters in a row, at least 4 of them will appear as ASCII in the output.

static ArraySlice<byte> Loyc.ByteArrayInString.ConvertToBytes ( string s )

static

Decodes a BAIS string back to a byte array.

Parameters

s	String to decode.

Exceptions

FormatException The string cannot be interpreted as a byte array in BAIS format.

Returns: Decoded byte array (use Convert(s).ToArray() if you need a true array).

static ArraySlice<byte> Loyc.ByteArrayInString.ConvertToBytes ( UString s )

static

Decodes a BAIS string back to a byte array.

Parameters

s	String to decode.

Exceptions

FormatException The string cannot be interpreted as a byte array in BAIS format.

Returns: Decoded byte array (use Convert(s).ToArray() if you need a true array).

static ArraySlice<byte> Loyc.ByteArrayInString.TryConvertToBytes ( UString s )

inlinestatic

Decodes a BAIS string back to a byte array.

Parameters

s	String to decode.

Returns: Decoded byte array, or null if decoding fails.

static ArraySlice<byte> Loyc.ByteArrayInString.TryConvertToBytes ( string s )

inlinestatic

Decodes a BAIS string back to a byte array.

Parameters

s	String to decode.

Returns: Decoded byte array, or null if decoding fails.

Remarks

Static Public Member Functions

Member Function Documentation