TimescaleDB's Hypercore Engine Squeezes Time-Series Data to 2% of Original Size

TimescaleDB claims compression ratios of up to 98% for typical time-series data - that's 50x to 100x savings on storage. For sensor floats that PostgreSQL's TOAST can't touch, hypercore makes them shrink by 10x to 20x. And timestamps? Delta-of-delta encoding pushes compression to 50x or 100x when intervals are regular.

Why TOAST Fails on Time-Series Data

PostgreSQL's TOAST (The Oversized-Attribute Storage Technique) only triggers when a single value exceeds roughly 2 KB. It compresses individual large values like long strings or JSON blobs using pglz or LZ4. For fixed-length types like timestamps and floats, TOAST does nothing - typical ratio: 1.0x. TimescaleDB's hypercore engine solves a different problem: cross-row patterns in time-series data. Where TOAST treats each value as an opaque byte stream, hypercore exploits numeric structure, monotonicity, and repetition.

Hypercore's Columnar Batch Compression

Hypercore is a hybrid row-columnar engine. New data lands in standard Postgres row-based chunks for fast INSERTs and UPDATEs. Older chunks get converted to a columnar compressed format automatically (e.g., older than 7 days). Each compressed chunk groups rows into batches of up to 1000 rows. A batch becomes a single row in the compressed table, with each column stored as an array of up to 1000 values. Analytical queries read only the columns they need, not entire rows. The column-major layout inside each batch enables efficient scans and advanced per-column compression algorithms.

How Delta Encoding and RLE Deliver 50-100x Compression

TimescaleDB applies multiple specialized algorithms per column. For numeric values that change by small amounts, delta encoding stores only the difference from the previous value. If the time interval is constant, delta-of-delta encoding yields zeros that compress to a tiny number of bits. For columns like machine_id or sensor_type where the same value repeats many times, run-length encoding (RLE) stores the value once plus a count. Gorilla XOR compression handles floating-point sensor data efficiently. The combination achieves 10-100x compression ratios on typical IoT workloads, where TOAST delivers none.

For teams storing years of sensor data, hypercore makes long retention economically viable without switching away from PostgreSQL.

Source: How TimescaleDB compresses time-series data
Domain: roszigit.com