INT96: the Parquet timestamp that refuses to die

Somewhere in your data lake there is a Parquet file with an INT96 column. The format was deprecated years ago, Spark and Impala wrote it by default for years anyway, and pipelines keep inheriting it. Mine did - large-scale Glue ETL jobs where a mis-parsed timestamp column doesn't crash anything, it just quietly writes wrong data. Which is worse.

Twelve strange bytes

INT96 is two numbers wearing one trench coat:

 ┌────────────────────────────┬──────────────┐
 │ 8 bytes                    │ 4 bytes      │
 │ nanoseconds since midnight │ Julian day   │
 └────────────────────────────┴──────────────┘
          both little-endian

The day count is a Julian day number - an astronomy epoch counting days since 4713 BC - and the time-of-day is nanoseconds, giving the format its one genuine virtue: nanosecond precision.

The trap

Both fields are little-endian, and INT96 values usually reach a human as a hex string read left to right. Parse 0x8096980000000000E8030000 big-endian and nothing errors - you just get a date in the wrong millennium. The failure mode of this format is plausible nonsense, which is precisely the failure mode data validation is worst at catching.

In .NET there's a second trap stacked on top: 12 bytes doesn't fit any primitive integer, so the natural tool is System.Numerics.BigInteger - which is signed and interprets your bytes' sign bit unless you handle it.

The parser

BigIntCsharp is the single-file, zero-dependency parser I wanted to find and didn't: hex-string and BigInteger entry points, explicit byte reversal for each field, Julian day converted through the standard Gregorian algorithm, and range validation so malformed input fails loudly instead of producing a believable wrong date.

var parser = new Int96TimestampParser();
DateTime ts = parser.ParseInt96Timestamp("0x8096980000000000E8030000");
// → validated, endianness-corrected, nanosecond-precise DateTime

The general lesson

Deprecated formats don't disappear; they fossilize into infrastructure. The engineering answer isn't to wish the data away, it's to make the handling boring: one file you can read in ten minutes, validation that refuses garbage, and tests around the two places (endianness, sign) where this format actually bites. Deprecation is a suggestion; data is forever.