Ingot · Open-source engineering project

Local time-series storage for Go applications.

Ingot is an alpha Go library for applications that need compressed metric history without a separate database server. The source, tests, benchmarks, and design notes are public.

Why an embedded database

Some Go applications need to keep sensor readings or metrics locally but cannot justify running Prometheus, InfluxDB, or another service. Edge gateways, self-instrumenting binaries, and offline tools still need durable writes, compression, retention limits, and time-range queries.

Ingot is a library rather than a monitoring platform. The host application remains responsible for synchronization, presentation, and operating policy.

My role
Architecture, implementation, tests, tooling, and documentation
Language
Go
Model
Label-indexed float time series
Evidence
Public source, design document, and validation suite

How the storage path works

Append→Write-ahead log→Memory head→Immutable blocks→Compaction and retention

Handle damaged writes explicitly

Committed samples pass through a segmented write-ahead log with checksummed records before entering the in-memory head. Recovery scans valid records and stops cleanly at a truncated or corrupt tail instead of accepting a partially written record.

Use one query path across memory and disk

Recent samples remain in memory while older data moves into immutable, memory-mapped blocks. Queries resolve label matchers and merge matching series across both layers, including the boundary where persistence and concurrency bugs tend to appear.

Bound storage as part of the design

Background compaction merges blocks into larger time spans and expires data according to policy. The design bounds steady-state memory by the active head and bounds disk by retention, while acknowledging temporary duplication during compaction.

Deliberate limits

Compression depends on the workload

Ingot uses Gorilla delta-of-delta timestamp and XOR value encoding. Benchmarks in the repository range from 0.42 bytes per sample for constant values to about 7.5 for an adversarial full-precision random walk. Compression varies with the data, so there is no single ratio that describes every workload.

Keep the database narrow

Replication, a query language, arbitrary value types, multi-process access, deletes, and out-of-order ingestion are explicit non-goals. Leaving them out keeps the storage model and failure surface appropriate for a single-process embedded library.

Make diagnosis possible

The repository includes tools for listing blocks, decoding chunks, checking index and checksum integrity, exposing database statistics, and serving a small HTTP query bridge. Stored data can be inspected directly instead of leaving diagnosis entirely to application logs.

What I test

The test suite covers damaged writes, concurrent access, unusual floating-point values, query correctness, and sustained load.

Recovery
Truncated and corrupt WAL-tail cases
Concurrency
Race testing across append and query behavior
Encoding
Round trips, adversarial values, fuzzing, and benchmarks
Queries
Comparison with a naive reference implementation
Resources
48-hour simulated sustained-load soak test

Ingot has not yet established reliability through external production use.