Rust CI/CD multi-arch build optimisation

Postgres, Lakekeeper and why we added the first Rust microservice

At EDB I work on a cross-cloud product called Postgres AI. This combines management of multiple Postgres clusters with AI tools such as Langflow, analytics and big data lakehouse functionality.

The product's backend consists mainly of Go microservices, with some Python services for AI-related capabilities such as MCP servers.

For the lakehouse, Apache Iceberg acts as the open table format, storing data in optimized columnar files such as Parquet in object storage. Lakekeeper serves as the REST catalog, tracking table metadata, schemas and partition snapshots.

Postgres powers this architecture as the transactional database storing Lakekeeper's metadata state, while EDB Postgres clusters query the external Iceberg tables natively. Together, they turn transactional Postgres databases into an open lakehouse that can query large analytical datasets alongside operational relational data.

Lakekeeper is written in Rust and uses iceberg-rust to manage Iceberg catalogs.

My team's core work covered user management and the control plane for Postgres AI. We had already used Rust to create WASM filters for Envoy, handling Dex IdP authentication and authorization at the edge. As the team with that Rust experience, we took on the work of adding Iceberg catalogs through a Lakekeeper microservice.

The difference between those two uses of Rust became important immediately. Our filters produced a single WASM target. Lakekeeper produced a native Linux executable, and the product needed container images for both amd64 and arm64.

Emulation and the existing multi-architecture build

The platform team provided a single standard Docker Buildx pipeline for building, scanning and publishing multi-architecture images. Buildx made it convenient to request both platforms from one workflow. When the host architecture did not match the target, QEMU provided CPU emulation.

That abstraction was useful for many services, but it hid an expensive detail for Lakekeeper. The arm64 image was not merely running an arm64 application under emulation. Its Docker build was running the Rust compiler, LLVM, the linker and native dependency build scripts under emulation.

Compiler workloads are particularly unfriendly to this arrangement. They are CPU-intensive, highly parallel and heavy on memory mapping, file system access, process creation and linking. Lakekeeper also pulled in native C, C++ and assembly-backed dependencies, so replacing QEMU with a simple pure-Rust cross compile would have required time consuming maintenance of a more complicated cross-compilation tool chain.

The size of the penalty varies with the workload, but independent reports give some useful context. 

One QEMU user-mode benchmark ran the same CPU-bound Go program about 6 times slower as an emulated arm64 binary than as a native amd64 binary. A Node.js CI case reported npm ci running 15 times slower under arm64 emulation.

So why hadn't QEMU use already caused a problem?
Because we don't compile Go using QEMU or run Node.js with it.

Our Go services avoided the penalty because Go can produce a binary for another operating system and architecture directly with GOOS and GOARCH, provided the dependency graph permits it.

Typescript just compiles to architecture independent Javascript run by the Node.js arch specific runtime.
It never needs to be run via QEMU to build the Docker images.

Python tends to be much less affected due to not having such CPU intensive compilation. Pip install of most wheels can just pick the pre-compiled source for the correct architecture and dodge this work too.  

Docker itself recommends native nodes or cross-compilation for compute-heavy builds. 

This explained an important clue in the timings. Rust itself was not taking two hours everywhere. The same source built in five to seven minutes locally. Upon analysis the huge difference appeared only in the architecture-emulated CI path.

So QEMU compilation wasn't needed for Typescript, was only slightly slower for Python and avoided for Go.  Our existing Rust components also avoided it because WASM was their deployment target.

Lakekeeper was the first service that made native Rust compilation part of our multi-architecture image pipeline and it hit a wall.

Rust CI/CD builds took almost 3 hours! the bulk of which was compilation under emulation.
Something that took under 10 minutes on real hardware.

Measuring the obvious caching fix

Before redesigning the workflow, I tested the apparent easy answer: cargo-chef.

cargo-chef helps Docker builds cache Rust dependencies separately from application source. It is a useful tool when dependency compilation is being repeated because a source change invalidates a Docker layer. Upstream Lakekeeper used it, so there was a reasonable case for trying the same approach.

I measured it rather than assuming it would help. The baseline build took 748 seconds. The build using cargo-chef took 757 seconds.

That result did not mean cargo-chef was ineffective in general. It meant dependency-layer hygiene was not the limiting factor in this build. The extra recipe and cook steps did not compensate for the cost of performing the compiler workload under emulation. Optimising the cache could not fix where the compiler was running.

Using Rust equivalents to GOARCH, such as cross, failed because they require pure Rust and we had a messy bag of C and assembly dependencies.

This was the point at which the problem changed from "how do we make this Dockerfile faster?" to "why are we compiling inside the multi-architecture Docker build at all?"

Separate compilation from image packaging

The replacement workflow moved native compilation out of the Buildx stage. Each binary was compiled on a GitHub Actions runner with the same CPU architecture as its target:

• the amd64 binary on a native amd64 runner;
• the arm64 binary on a native arm64 runner.

The two jobs ran in parallel. Neither needed QEMU.

After compilation, a small packaging Dockerfile copied the prebuilt executable into the existing runtime image. The workflow published one image for each architecture and then combined them under a multi-architecture manifest.

The design changed the location of the expensive work without changing the artefact expected by the rest of the platform:

    native amd64 runner ----> amd64 binary ----> amd64 image ---+
                                                                |
                                                                +--> multi-arch manifest
                                                                |
    native arm64 runner ----> arm64 binary ----> arm64 image ---+
                                      |
                                      +--> existing scans and deployment rendering

It would be slightly misleading to call this "removing Docker from the build." Containers still had an important role. We separated native compilation from multi-architecture image packaging, rather than asking Buildx and QEMU to act as a cross-architecture Rust build system.

The build environment is part of the binary contract

The first native version revealed another problem. I initially compiled on an Ubuntu 24.04 environment, but the production runtime image was based on UBI9. The resulting executable required glibc 2.38 and 2.39 symbols that were not available in the older runtime environment.

The build was fast, but the binary could not run in the intended image.

That failure clarified the real compatibility requirement. Matching the CPU architecture was only half of the problem. A dynamically linked native binary also carries expectations about its userspace ABI (application binary interface).

I changed the workflow so that compilation ran in a Redhat minimal UBI9 build container on each native runner. That kept execution native while aligning the build environment with the runtime image's glibc ABI. We retained the reproducibility and dependency boundary of a containerized build without emulating the target processor.

This became one of the most useful lessons from the work: the build image and runtime image form a compatibility contract. A successful compiler exit code does not prove that the artefact can run in the environment where it will be deployed. See the full deploy and test details below.

Keep the security and release path intact

A fast side pipeline would have been easy to create. It would also have been the wrong solution.

The existing platform workflow did more than compile an executable. It owned image tags, registry publication, multi-architecture manifests, deployment rendering and security checks. Replacing that entire path with a Lakekeeper-specific script would have duplicated platform policy and made the fast build harder to maintain.

Instead, the new path changed only the compilation and packaging stages. It continued to feed the resulting image into the existing Anchore image scan and TruffleHog secret scan. The same image digest flowed into the normal Helm and Kustomize rendering and registry publication steps.

That constraint shaped the implementation. Performance work on a release pipeline is incomplete if it quietly removes the controls that made the old pipeline trustworthy.

I also implemented the native Rust path as a reusable workflow and proposed it for the platform team's shared build tooling. Lakekeeper was the first service that needed it, but it was unlikely to be the last native Rust service. A future repository should be able to provide its build inputs as a thin caller, not copy and gradually diverge from a bespoke workflow.

The result

The controlled end-to-end comparison was:

Measurement	| Original workflow	| Native workflow
Full build	1h 48m 28s	9m 19s
Relative improvement		about 11.6x

Other runs of the old workflow had taken as long as 2 hours 48 minutes, but I use the 1 hour 48 minute run for the speedup calculation because it was the controlled comparison rather than selecting worse historical examples.

With warm caches, the native Cargo stages took roughly 3 minutes 55 seconds on amd64 and 4 minutes 40 seconds on arm64. Because they ran concurrently, the workflow paid approximately the slower of those two costs rather than their sum. The remaining time covered packaging, publication, manifest assembly, scanning and deployment rendering.

When making such changes it is essential to deploy the built artefacts, ie the new pipeline's lakekeeper microservice images to the Postgres AI ephemeral cluster and fully test it.
There can be image comparisons made that validate an artefact is effectively the same as one built with the old pipeline, but the safest test is to fully exercise it in a near production deployment.
Especially if you have all the test build, CI/CD and test suites in place already for refactor validation.

I deployed the resulting image to a real ephemeral arm64 Kubernetes cluster. The service started with both containers ready and no restarts, its PostgreSQL migrations completed, and its control-plane and service-mesh connections succeeded.

I then ran the full REST functional suite. It passed all 41 tests in 144.3 seconds with no failures. Those tests exercised Iceberg catalog operations, Postgres lifecycle behaviour and the authorization matrix, rather than merely checking that the process returned a health response.

What I would carry into the next build pipeline

Four principles from this work apply beyond Rust and Lakekeeper.

First, measure the proposed optimisation. Caching was the obvious answer, and in this case it made no measurable improvement. The experiment prevented us from fixing the wrong layer of the system.

Second, treat architecture-specific compilation as a scheduling problem. If native workers are available, moving the compiler to matching hardware can be simpler and safer than maintaining emulation or a bespoke cross-toolchain for a large native dependency graph.

Third, make the build and runtime ABI relationship explicit. Matching amd64 or arm64 does not guarantee that a Linux binary will run in a given Linux image. The libc and native library boundary matters too.

Finally, optimise inside the release contract rather than around it. The most valuable part of the solution was not only reducing the build from nearly two hours to nine minutes. It was doing so while retaining the same scans, publication rules, image format and deployment path used by the rest of the platform.

The original pipeline made multi-architecture builds look uniform, but the workloads underneath were not uniform. Once Lakekeeper introduced Hybrid Manager's first native Rust service, QEMU turned that convenient abstraction into a two-hour tax on every change. The durable fix was to stop treating an emulated Docker build as the only place compilation could happen, put each compiler on the architecture it targeted, and leave the rest of the secure delivery system intact.