Encoder Internals

The compile function in encoder.ts performs a two-pass encoding.

Pass 1: Size Calculation


let totalSize = HEADER_SIZE; // 4 bytes
const fieldBuffers: Array<{ fieldId: number; typeTag: TypeTag; data: Uint8Array }> = [];
 
for (const { field, value } of fieldsToEncode) {
  const data = encodeValue(value, typeTag, field.name);
  fieldBuffers.push({ fieldId: field.id, typeTag, data });
  totalSize += FIELD_HEADER_SIZE + data.length; // 4 + data
}

Each field value is encoded into a temporary Uint8Array. The total size is accumulated to allocate a single buffer in pass 2.

Pass 2: Binary Write


const buffer = new ArrayBuffer(totalSize);
const view = new DataView(buffer);
const bytes = new Uint8Array(buffer);
let offset = 0;
 
// Header
bytes[offset++] = 0x4C; // 'L'
bytes[offset++] = 0x44; // 'D'
bytes[offset++] = schema.version;
bytes[offset++] = fieldBuffers.length;
 
// Fields
for (const { fieldId, typeTag, data } of fieldBuffers) {
  bytes[offset++] = fieldId;
  bytes[offset++] = typeTag;
  view.setUint16(offset, data.length, false); // big-endian
  offset += 2;
  bytes.set(data, offset);
  offset += data.length;
}

The two-pass approach avoids buffer resizing and produces a single contiguous allocation.

Value Encoding

String

Direct UTF-8 via TextEncoder.encode(). No length prefix in the value data — the field header’s 2-byte length covers it.

Bool

Single byte: 0x01 or 0x00.

String Array


[count: uint16] [len: uint16] [bytes...] [len: uint16] [bytes...] ...

Map

Keys are sorted with Object.keys(value).sort() before encoding:


[count: uint16] [klen: uint16] [key...] [vlen: uint16] [val...] ...

The sort ensures two maps with the same entries (in any order) produce identical bytes.

Constants


const MAGIC = [0x4C, 0x44] as const; // "LD"
const HEADER_SIZE = 4;               // magic(2) + version(1) + count(1)
const FIELD_HEADER_SIZE = 4;         // field_id(1) + type_tag(1) + length(2)