Faro

The constraint

I run a dozen services across two VPS providers. Hosted APM would have cost more per month than the servers. The open-source alternatives were either a single-tenant toy or a five-container stack that needed its own operator.

I wanted one binary, one database, and per-tenant separation good enough to point a client at.

What I chose

ClickHouse, and shaping the schema around the query I actually run. Observability data is write-once, read-by-range, and enormously compressible. The interesting decision wasn’t picking ClickHouse — it was the sort key. Traces are ordered by (tenant, service, timestamp) because the real question is always “what was this service doing during this window,” never “find me this trace id” first.

Rust on the ingest path. Ingest is the only part under genuine pressure: it runs constantly, its cost is pure overhead, and a garbage-collection pause there shows up as a lie in someone’s latency graph. Axum plus a batching writer, with backpressure that drops rather than queues without bound. Losing a span is acceptable; taking down the thing you use to debug outages during an outage is not.

OTLP in, nothing custom. Speaking the OpenTelemetry protocol meant every language I use already had an instrumented client. I wrote zero SDKs.

On naming

Faro is Spanish for lighthouse. It seemed right for a thing whose entire job is to be the fixed point you navigate by when everything else is dark and moving.

What I would do differently

I shipped the query layer before deciding on a retention policy, so early tenants have unbounded historical data and a materialised-view strategy bolted on afterwards. Retention is a schema decision, not a settings screen.