Package a reusable dataset

1. Choose the published domain

A dataset needs a time interval that every stored series can support. Requested input coverage is not enough: complete segments can drop a remainder, and retry attempts may change their length. We must compare the actual fitted ends before admitting a chart time.

Suppose all fitted series begin at day 10 and their actual ends are day 74, day 78 and day 80. The shared fitted interval is [10, 74], because the earliest end limits what all series can provide. The builder then moves the low endpoint inward by five days and the high endpoint inward by one day, publishing [15, 73]. These invented offsets are not UTC dates.

fitted_hi = minimum of accepted series ends

domain_lo = fitted_lo + 432000 s

domain_hi = fitted_hi − 86400 s

This narrower interval is the published domainPublished domainClose The interval in which the runtime admits chart epochs. It is narrower than the shared fitted support, leaving stored coefficients available for nearby calculations. Jet Propulsion Laboratory: Navigation and Ancillary Information Facility — ephemeris coverage and polynomial records. Runtime admission uses it, while nearby calculations can use the stored fitted support. At the public endpoints in our example, centered speed calculations sample half a day away, at days 14.5 and 73.5. Both lie within the shared fitted support.

The current builder gives every series the same fitted start lo and takes the smallest actual end among the accepted ladder results. This is a second coverage decision: earlier we intersected source kernels and reserved one day on each side before fitting. Now we intersect what was actually fitted. Individual series can retain more support than the common fitted interval.

const publishedDomain = (fittedLo: number, fittedEnds: number[]): [number, number] =>
  [fittedLo + 5 * 86400, Math.min(...fittedEnds) - 86400];

2. Write the runtime format

The runtime no longer needs the preparation machinery once the fitted angles are stored. It needs a way to find each series and read its coefficients. A binary file supplies that arrangement: a fixed header describes the dataset, a table describes the series, and the coefficient arrays follow in table order.

Start with an invented two-series file. Its longitude series has two degree-1 segments, giving four coefficients. Its declination series has two degree-0 segments, giving two coefficients. The header occupies 64 bytes and each table entry occupies 48, so coefficients begin at 64 + 2 × 48 = byte 160. Each coefficient takes eight bytes: longitude occupies [160, 192) and declination [192, 208). Total length is 208 bytes. An interval [a, b) includes byte a through b − 1. Production has 25 series; this small file teaches offsets.

table_end = 64 + 48m

coefficient bytes = 8 × sum over series of (segments × coefficients per segment)

total length = table_end + coefficient bytes

Here m is the number of series. Offsets count bytes from zero. The 64-byte header begins with eight ASCII bytes spelling HDCHEB01. At byte 8 is a four-byte table CRC, and at byte 12 a four-byte series count. Bytes 16, 24, 32 and 40 begin eight-byte public low/high and fitted low/high times. Byte 48 begins the eight-byte total length. Byte 56 holds a four-byte coefficient CRC, and byte 60 the four-byte engine fingerprint.

Each 48-byte series entry stores body ID at entry offset 0 (two bytes), quantity ID at 2 (two), coefficients per segment at 4 (four), t0 at 8 (eight), segment length at 16 (eight), segment count at 24 (four), and coefficient byte offset at 32 (eight). The writer leaves bytes 28–31 and 40–47 zero. Body IDs are local enums rather than NAIF IDs; NodeOmega is 12. Quantity 0 means longitude and 1 means declination.

An integer such as 25 is written as four bytes [25, 0, 0, 0], with the least significant byte first. This is little-endian byte orderLittle-endian byte orderClose Store the least significant byte first. For integer 25 in a four-byte field, the bytes are [25, 0, 0, 0]. HDCHEB01 uses this order for all its numeric fields. RFC 1952 (1996), Deutsch: CRC-32 sample algorithm and byte order. All numeric fields follow that order. Times and coefficients use eight-byte IEEE-754 floating-point values, preserving their fractional values rather than storing text or integer-rounded degrees. A series with n segments and k coefficients per segment consumes 8nk coefficient bytes.

The writer also calculates a cyclic redundancy check (CRC)Cyclic redundancy checkClose A checksum computed from bytes using bit operations. CRC-32 produces a 32-bit result and detects many accidental changes. It is not an authentication mechanism. RFC 1952 (1996), Deutsch: CRC-32 sample algorithm and byte order, a compact check for accidental byte changes. Its coefficient checksum, named full_crc32, covers only bytes from table_end through the file end. After writing that field and the fingerprint, it computes the table CRC over [12, table_end), including the remaining header fields and series entries. The magic and table-CRC field are outside that range.

To understand the checksum loop, take the one-byte input [255]. CRC-32 begins at 0xFFFFFFFF. XOR with 255 flips the low eight bits, leaving 0xFFFFFF00. Those eight low bits are zero, so eight right shifts leave 0x00FFFFFF without applying the polynomial XOR. Complementing all 32 bits gives 0xFF000000, or 4278190080. This deliberately simple byte lets us follow the complete loop by hand.

For a general byte, XOR it into the accumulator and repeat eight times: shift right one bit, then XOR 0xEDB88320 if the old low bit was 1. Complement the final result. Hexadecimal writes four bits per digit. XOR flips bits wherever its other operand is 1. With other bytes an old low bit can be one, so the polynomial operation is needed.

In the teaching TypeScript, ^ means XOR, & selects shared 1 bits, >>> shifts right with zero bits entering on the left, and ~ flips every bit. The expression c & 1 selects the old low bit: 1 for odd integers, 0 for even ones. For an unsigned integer, one right shift is floor(c/2), dropping that bit. The final >>> 0 interprets the result as an unsigned 32-bit integer instead of a negative signed value.

RFC 1952 includes the reflected CRC-32 convention and sample code used to explain this calculation. HDCHEB01 is a repository format rather than a gzip stream. CRC detects many accidental changes but does not authenticate a dataset. SHA-256, discussed next, is a separate cryptographic hash with a different role.

const coefficientBytes = (segments: number, degree: number): number =>
  8 * segments * (degree + 1);
const crc32 = (bytes: readonly number[]): number => {
  const state = bytes.reduce((acc, byte) => {
    let c = acc ^ byte;
    for (let bit = 0; bit < 8; bit += 1)
      c = (c >>> 1) ^ ((c & 1) ? 0xEDB88320 : 0);
    return c;
  }, 0xFFFFFFFF);
  return (~state) >>> 0;
};

3. Record and check the resulting artifact

The same filename can hold different bytes, so a handoff needs more than a path. We also need to identify the inputs, the builder executable and the output, and record which checks completed. That record lets a host follow the build that produced the dataset.

A cryptographic hashCryptographic hashClose A fixed-length value computed from input bytes. This workflow uses the full SHA-256 values to identify artifacts and compare them with recorded values. An identity check does not itself establish astronomical correctness. National Institute of Standards and Technology: FIPS 180-4 (2015), Secure Hash Standard produces a fixed-length identifier from bytes. The workflow uses SHA-256, a 256-bit member of the Secure Hash Algorithm family, to identify its full inputs and outputs. For the engine fingerprint inside HDCHEB01, it takes only the first eight hexadecimal characters of the executable hash. Each hex digit represents four bits, so this shortened identifier contains 32 bits, or four bytes.

For an invented executable hash beginning 0000002a…, hexadecimal 2a is 2 × 16 + 10 = 42. The writer stores fingerprint 42 in little-endian bytes [42, 0, 0, 0]. This identifies a prefix of the executable hash; it is not the full cryptographic identity of the executable or all build inputs.

fingerprint bits = 8 hex digits × 4 = 32 bits

Before building, regenerate.py requires complete acquisition and verifies each recorded input SHA-256. It refuses an existing output directory. It rebuilds the two intermediate asteroid kernels from the recorded responses, compares them with cached Horizons spot responses, and applies their one-arcsecond gate. It then invokes the native angular builder, supplying the fingerprint derived from the executable hash.

The builder writes the blob, residual report and manifest, including the blob SHA-256. The orchestrator hashes cheb.bin again and compares it with manifest.sha256. Next it invokes examples/cloudflare-worker/verify.mjs with the new dataset and a wasm-validation.json report path. Only after that checked command succeeds does it write provenance.json with usable: true, input metadata, output hashes, executable hash, lockfile hashes, command and environment.

usable record follows input checks → kernel gates → angular gates → blob hash check → local Wasm command success

Suppose the blob hash matches its manifest but the local Wasm command fails. This invocation does not write usable: true, even though intermediate files may remain. A file or manifest alone does not establish that the usable record was written. Matching hashes identify bytes; they do not prove that those bytes produce correct charts.

Carry the manifest and provenance alongside the exact dataset bytes when handing them to a host. Reading and checking these files is an effect at the host boundary. The chart runtime accepts caller-supplied bytes; it does not download data, choose storage or regenerate a dataset. SHA-256 is specified by the National Institute of Standards and Technology’s Secure Hash Standard; this workflow’s usable flag and check order are repository decisions.