The Invisible Tax — Telemetry Pipeline Conversion Costs — KubeCon Mumbai 2026 Deep Dive 17

Deep dive 17 closes the series. Cijo Thomas (Microsoft) — who works across OpenTelemetry language implementations (Rust, .NET), the Specification, and OTel Arrow, on the team behind Microsoft's planetary-scale telemetry pipelines — names a cost almost nobody budgets for: the CPU your telemetry pipeline burns just converting data between formats at every boundary. The wire format (OTLP protobuf) is only a wire format — it's nobody's in-memory home — so every SDK, every collector hop, and every backend decodes it, walks it record by record, and re-encodes it. The fix is a shared, columnar, language-neutral in-memory format. OTel is already building it: Apache Arrow via OTAP (OpenTelemetry Protocol with Apache Arrow) and a new Arrow-native collector runtime, the Dataflow Engine. Early result: ~20× throughput on the same cores, with transforms that are nearly free.

It's the perfect bookend to the observability block. DD14 harvested mesh telemetry for free and DD03 (Kafka observability) warned about cardinality blowing up cost. This talk goes one level deeper — below what you collect, to how the bytes move — and finds a structural tax hiding in plain sight.

Why talk about pipelines at all?

Telemetry volume keeps growing, and the industry has fought back hard on two fronts: storage (tiered, columnar backends) and what gets ingested (sampling, filtering, drop rules). Both are well-trodden. The pipeline itself — the path between the app and the backend — gets far less attention. That gap is the whole subject.

A pipeline is, precisely: anything between the moment a telemetry signal is produced and the moment it lands in a queryable state in some backend. Along that path it does real, useful work — transport, batch & compress, filter & sample, enrich, redact, route, transform. None of that is wasteful. The waste is somewhere else.

Pipelines cost CPU — a lot of it

Every one of those operations runs on real CPU: in collectors, in sidecars, in regional aggregators, on every host. (Memory and network too, but CPU is usually the loudest.) At scale, pipeline CPU adds up fast — running collectors at every hop, on every host, becomes a genuine line item in the observability bill. So the question worth asking is: where does that CPU actually go?

The setup for the whole talk. The instinct is to blame the processors — the filter rules, the transforms, the enrichment. Cijo's argument is that those are cheap. The expensive part is the machinery that runs around every processor on every batch: decode, allocate, walk, re-encode, garbage-collect. That machinery is invisible because it's not a feature you configured — it's overhead you inherited.

How cheap is the simplest possible operation?

Take the most trivial transform imaginable: rename an attribute key on a log record — exception.type → exception.kind. Everything else about the record stays identical.

The measured cost of the rename itself: ~30 nanoseconds per record. A single CPU core, in a tight loop, can do roughly 30 million renames per second. So the actual rename cost is negligible.

And yet — anyone who has run a real telemetry pipeline knows a single core gets nowhere near 30 million records per second. The rename is free; something around it is not. That gap is the invisible tax.

What actually goes on inside a collector

For every batch the pipeline receives, three things happen:

Decode the protobuf bytes → allocate a full in-memory object graph.
Run any configured processors (filter, transform, route…).
Re-encode back to bytes — and throw the whole object graph away.

The tax is paid even with zero processors. Run a collector in pure passthrough — no filters, no transforms — and decode + re-encode + GC churn still happen on every batch, at every hop. The pipeline is doing nothing useful and still burning CPU. Add a single rename rule and throughput drops further, CPU climbs. The conversion happens once per batch, so the extra CPU per rule isn't the conversion — it's that the in-memory shape itself is expensive to walk, record by record.

This is the crux: the row-oriented object graph you get from decoding OTLP protobuf is costly to traverse, and every per-record processor pays that traversal cost again.

Fig 1 — The OTLP pipeline pays decode → allocate → walk → re-encode → discard on every batch at every hop. The conversion at the boundary, not the work, is the tax.

Your application pays this too

It isn't only collectors. Anything emitting telemetry — through any OTel SDK — runs the same shape conversion before bytes leave the process:

SDK record  →  protobuf struct  →  byte[]

That's two conversions per record, on every host emitting telemetry. Cijo put real numbers on the per-log-record cost across language SDKs:

SDK	Conv 1 (SDK → struct)	Conv 2 (struct → bytes)	% in Conv 1	Total
Rust	395 ns	114 ns	~77%	~510 ns
Go	281 ns	598 ns	~32%	~887 ns
.NET (skips Conv 1)	—	—	0%	~194 ns

The .NET result is the proof. .NET emits the protobuf struct directly from the SDK, skipping the first conversion entirely — and lands at ~194 ns, several times cheaper than Rust or Go. That's evidence that Conversion 1 is avoidable, not free. The cost isn't inherent to producing telemetry; it's an accident of each component choosing its own in-memory shape.

This is a structural problem, not a tuning problem

The collector and the OTel SDK pay the same three costs: conversion at every hop, GC churn from per-batch allocations, and per-row processing. The root cause is singular: every component picks its own in-memory shape, so conversion happens at every boundary — not by design, but by accident.

OTLP protobuf is only a wire format. It's nobody's in-memory home. It's nested and opaque: every hop has to decode it before it can do anything, and once decoded it's walked one record at a time. You cannot tune your way out of that — no flag makes a nested, row-oriented format cheap to traverse in bulk. It needs a different shape.

What would the fix look like?

Wire transport is unavoidable — bytes have to cross processes and machines. So the goal isn't to eliminate the wire; it's to choose an in-memory representation that:

Maps cheaply to/from the wire — minimal decode and encode work at boundaries.
Fits how pipelines actually work — bulk, per-column operations over batches, not per-record walks.
Is language-neutral — an SDK in Rust, a processor in Go, and a backend in Java all share the same shape with no conversion.

Good news, said Cijo: OTel is already building on exactly such a format.

Apache Arrow, OTAP, and the Dataflow Engine

Piece	What it is
Apache Arrow	A columnar, batch-oriented in-memory format. The same layout on the wire and in memory — so "decode" is essentially a pointer cast.
OTAP	OpenTelemetry Protocol with Apache Arrow — a 100%-compatible OTLP alternative that carries telemetry as Arrow record batches.
Dataflow Engine (DFE)	A new collector runtime that uses OTAP as its in-memory representation, so processors operate directly on Arrow batches.

Together they deliver the three properties above: no conversion at boundaries, no per-record allocation, columns instead of rows.

Why columns change everything. In OTLP, a log batch is a nested tree: ResourceLogs → ScopeLogs → LogRecords, each with its own attributes, walked node by node. In OTAP the same data is flat columnar tables — resource_attrs, scope_attrs, log_attrs, logs — joined by parent IDs. The wire layout equals the memory layout, so ingesting a batch is near-zero-copy, and a rename is one operation applied to a whole column at once rather than a walk over every record.

Fig 2 — Same telemetry, two shapes. OTLP is a nested tree decoded and walked per record; OTAP is flat Arrow columns where the wire layout is the memory layout.

Same runtime, only the protocol changes

The cleanest experiment: take the same Dataflow Engine, on the same cores, and change only the wire format:

Wire protocol	Throughput @ 1 core
OTLP in/out (decode + convert at the boundary)	121K logs/s
OTAP in/out (Arrow end to end)	2.47M logs/s — ~20×

Twenty times the throughput, same engine, same hardware — the entire difference is paying or not paying the conversion tax at the boundaries.

Transforms become nearly free

Because Arrow was designed for bulk operations over columns, and a rename is exactly a bulk column op, adding rules costs almost nothing in OTAP:

Rename rules added	OTel Collector (OTLP)	DFE (OTAP)
+1	+3.75% CPU	+0.07% CPU
+2	+7.5% CPU	+0.14% CPU
+3	+11.25% CPU	+0.21% CPU

The OTLP representation walks each record, so every rule stacks per-row work. OTAP operates in columns — adding rules is essentially free. That's a roughly 50× difference in the marginal cost of a transform.

Where this is going — Phase 2 and Phase 3

Everything above is the result of OTel Arrow Phase 2: OTAP on the wire, the Dataflow Engine, and Arrow-native pipeline processing. But the team isn't done — Phase 3 is the more ambitious goal: telemetry born in Arrow.

Directions in scope for Phase 3:

SDKs emit Arrow batches directly — no protobuf step at the application (killing Conversion 1 entirely, as .NET already hints is possible).
Pipelines stay in Arrow end to end — no per-hop conversion anywhere.
Backends ingest Arrow — many analytics engines already speak it natively.
Pluggable components — run existing collector-contrib components on the new engine.

The vision: one representation, from source through every hop to query. Telemetry born in Arrow, live end-to-end in Arrow.

Honest caveat. This is early — incubation, not production. The ~20× numbers are benchmarks on a young runtime, not a drop-in upgrade you ship tomorrow. The point of the talk isn't "switch now"; it's that the conversion tax is real and structural, the fix is understood, and the interesting work is just starting.

FAQ

If renaming a field is 30 ns, why is my collector slow?

Because the rename isn't what costs you. Every batch is decoded from OTLP protobuf into a nested object graph, walked record by record, then re-encoded and discarded — even with zero processors. That decode/allocate/walk/re-encode/GC cycle is the tax, and it's paid at every hop.

Isn't protobuf supposed to be fast?

Protobuf is a fine wire format — compact and portable. The problem is using it as nobody's in-memory home: it's nested and opaque, so every component decodes it into its own object shape and re-encodes on the way out. The cost is the conversion at boundaries, not protobuf's encoding per se.

What is OTAP, and is it a breaking change?

OTAP (OpenTelemetry Protocol with Apache Arrow) is a 100%-compatible OTLP alternative that carries telemetry as columnar Arrow record batches. Because the wire layout equals Arrow's memory layout, ingesting it is near zero-copy. It's an additional protocol, not a replacement that forces a rewrite.

Why does columnar give ~20× and near-free transforms?

Arrow stores data as columns, so the engine skips per-record decode and operates on whole columns at once. Ingesting OTAP is near zero-copy (the ~20× throughput), and a transform like a rename is one bulk column operation instead of a per-record walk (so adding rules costs ~0.07% vs ~3.75% CPU each).

Can I use this today?

It's incubation, not production. OTel Arrow Phase 2 (OTAP + the Dataflow Engine) exists and produces these benchmarks; Phase 3 (SDKs and backends born in Arrow) is just starting. Follow github.com/open-telemetry/otel-arrow rather than swapping your production pipeline now.

Takeaways

The conversion tax is real, structural, and universal — it applies to every OTel component: SDKs, collectors, backends.
The work is cheap; the machinery around it is not — a rename is 30 ns, but decode/allocate/walk/re-encode/GC is paid on every batch, even in passthrough.
Root cause: every component picks its own in-memory shape, forcing conversion at every boundary by accident. .NET skipping Conversion 1 proves it's avoidable.
The fix is a shared, columnar, language-neutral in-memory format — Apache Arrow, via OTAP and the Dataflow Engine.
Early results: ~20× throughput on the same cores and transforms ~50× cheaper at the margin — but it's incubation, not production.
Like a currency conversion fee: invisible per transaction, painful at scale — and worth removing.

That's a wrap on Day 1. Seventeen deep dives — from re-architecting monoliths and shared-first platforms, through policy-as-code and federated identity, into agents, GPUs, and serving, and out to observability, the edge, and storage. The throughline: cloud-native is maturing from "can Kubernetes do this" into "how do we do it at population scale, under regulators, without burning the budget." Day 2 notes land later. Back to the index for the full table of contents.

References

KubeCon Mumbai 2026 — Day 1 index · the full 17-talk series
open-telemetry/otel-arrow · OTAP, the Dataflow Engine, benchmarks
Apache Arrow · the columnar in-memory format
DD14 — The Lean Observability Stack · harvesting mesh telemetry for free
DD03 — Kafka observability & cardinality · the other side of telemetry cost

The invisible tax — how data format conversions drive up telemetry pipeline costs.