Generated by Codex with GPT-5

What happened

Databricks’ official blog published Ingesting the Milky Way: Petabyte-Scale with Zerobus Ingest, a June 11, 2026 post about the architecture behind Zerobus Ingest, Databricks’ managed streaming ingestion service for writing high-volume producer streams directly into Delta tables governed by Unity Catalog.

The post is interesting because it is not just a launch note for a lakehouse ingestion API. Databricks uses a concrete benchmark, NASA’s NEOWISE dataset, to show a petabyte-scale write path sustaining roughly 12 GB/s into a single table over 24 hours. The engineering story is how the system avoids making customers pre-size brokers, pick partition counts, operate connectors, or keep Kafka-shaped infrastructure around when the intended destination is already the lakehouse.

The central move is to make streaming ingestion feel like a push API while still preserving the properties that made teams use durable message buses in the first place: ordering, low-latency handoff, backpressure-friendly acknowledgements, and scalable fan-in from many producers. Zerobus is Databricks’ attempt to move those responsibilities into a managed data plane rather than leaving every data team to design and tune the same ingestion stack.

The architecture

Traditional streaming systems usually couple two concerns into the same abstraction. A partition is both the unit of ordering and the unit of parallelism. That is powerful, but it creates long-lived operational consequences. Teams must estimate peak load up front, route producers to partitions, keep enough partitions to handle high watermarks, and be careful about changing the partition topology because consumers may depend on per-partition ordering.

Zerobus changes the contract. Instead of promising that a partition is ordered, it promises that a producer’s stream connection is ordered. When a producer opens a stream to the service, that stream becomes the logical unit whose records must arrive in order. Internally, Databricks can route new streams across a pool of pods using hot-routing heuristics. If one pod is under pressure, new streams can land elsewhere; if demand falls, pods can stop receiving new streams and drain existing ones before the pool shrinks.

That design matters because it separates user-visible semantics from infrastructure scaling. The producer does not need to know which pod, broker, partition, or worker owns the current load. It only needs the stream-level ordering contract. Databricks can then autoscale around stream count and throughput, using the connection as a finer-grained scheduling unit than a static partition assignment.

The second important choice is data handling. Zerobus has to ingest row-by-row streams without spending its budget on avoidable decoding, copies, or allocations. The post highlights Zeroparser, a custom protobuf decoder that parses directly from the wire format while still accepting dynamic protobuf descriptors. Rust’s lifetime system is part of the mechanism: the parser can borrow from the network-owned bytes safely, avoiding a decode path that materializes full intermediate objects just to convert them again for storage.

The third piece is durability. Zerobus implements a latency-optimized write-ahead log before publishing data into the lakehouse. Once records are durable, the server acknowledges progress by returning the highest committed offset on the stream rather than acknowledging each record individually. Over a gRPC bidirectional stream, producers can keep sending records on one channel while receiving committed offsets on the other. A client can then drop everything up to the acknowledged offset from its local in-flight buffer.

That acknowledgement model is a quiet but important part of the system. It gives producers a compact durability signal and keeps client memory bounded under high-throughput streaming. Databricks then uses Delta Kernel Rust for the core Delta-writing path, so the ingestion service’s durability and commit machinery connects directly to the storage format customers query later.

The benchmark

Databricks chose the NEOWISE dataset because it behaves like the kind of large telemetry stream Zerobus is meant to absorb: many observations, a continuous time axis, and enough volume to expose producer, network, ingestion, and table-writing bottlenecks. The benchmark used Locust on Kubernetes to simulate fan-in from many independent producers, rather than relying on one large machine that might benchmark the producer host instead of the service.

The reported test used 2,048 Locust workers, one Zerobus stream per worker, a slow ramp-up, 50,000 in-flight records per stream, and protocol-buffer binary messages. Over the 24-hour measurement window, Zerobus ingested about 1.04 trillion rows at roughly 11.8 GB/s of proto2 wire throughput into a single Unity Catalog table. Databricks emphasizes that the figures come from rows fully committed to Delta storage, not just bytes accepted at the edge.

The single-table detail is important. Many distributed ingestion claims become easier if the workload can be spread across many independent destinations. Zerobus is showing that the managed service can concentrate a very large producer fan-in onto one governed table without making customers expose or tune the partitioning machinery underneath.

Why it matters

The broader engineering takeaway is that managed streaming systems should hide operational topology without weakening semantic contracts. Kafka-style partitions made sense because they gave engineers a concrete way to reason about order and scale. But once every team has to forecast peak load, manage connector lifecycles, overprovision partitions, and handle partition-change consequences, the partition becomes both a tool and an accidental product interface.

Databricks’ design pushes that accidental interface down a layer. The product surface is a stream connection and a destination table. The platform surface is dynamic routing, pod autoscaling, zero-copy parsing, a write-ahead log, offset acknowledgements, and Delta commits. This is a cleaner split for teams whose real goal is not to run a message bus, but to make fresh telemetry, product events, or operational signals queryable with low delay.

There is also a lesson for AI and observability infrastructure. Agent traces, model telemetry, application events, IoT feeds, and security signals increasingly have the same shape: many producers, bursty volume, strict governance needs, and pressure to make the data available quickly for analysis. A managed ingest layer that writes directly into governed analytical storage can remove an entire class of glue systems between production services and downstream evaluation, monitoring, or incident response.

Takeaway

Zerobus Ingest is a useful example of rethinking streaming around the destination rather than the intermediary. If the end state is a governed Delta table, the ingestion system can own the broker-like responsibilities internally and expose a simpler contract to producers: open a stream, send ordered records, receive committed offsets, and query the table shortly after.

The strongest part of the Databricks post is that the architecture choices line up with the benchmark. Stream-level ordering enables dynamic partitioning and autoscaling. Zero-copy parsing protects the CPU and memory budget. The write-ahead log and offset acknowledgements preserve low-latency durability. The NEOWISE run then exercises those choices under large fan-in instead of presenting them as isolated optimizations.

The result is a systems pattern worth watching: durable ingestion as a managed part of the lakehouse data plane, not as a separate operational estate that every team must size, shard, and operate. For production engineering teams, the value is not only higher throughput. It is a simpler boundary between event-producing services and the analytical systems that depend on their data.