Skip to content

ADR-002: Use Pre-Built, Hermetic CI Build Images

Field Value
Date 2026-03-19
Status Accepted
Deciders Project owner
Supersedes —
Superseded by —

Context and Problem Statement

CI pipelines that install tools at runtime (e.g., apt install, go install, pip install) have three failure modes: they are slow (every job downloads packages), non-reproducible (a package update can silently change behaviour between two otherwise identical runs), and fragile (a transient network error or upstream package removal breaks the build).

The project needs a build environment that is fast, fully reproducible, and immune to external changes at build time.

Decision Drivers

  • Builds must be reproducible: the same commit must produce bit-identical artifacts on any day.
  • No outbound internet access is permitted during pipeline jobs (security and air-gap compliance).
  • Tool versions must be explicitly managed and auditable.
  • Build startup time must be minimal — no package downloads in the hot path.

Considered Options

  1. Install tools inside pipeline jobs at run time.
  2. Use a shared, mutable "builder" VM with tools installed ad-hoc.
  3. Use pre-built, version-pinned Docker images as the CI execution environment.

Decision Outcome

Chosen option: Pre-built Docker images (option 3), because it is the only option that satisfies all four decision drivers simultaneously.

Positive Consequences

  • Builds are hermetic: the image digest fully determines the tool set; no network calls needed.
  • Tool version changes are explicit: they require a new image build and a pipeline variable update, creating a clear audit trail.
  • Pipeline jobs start immediately — the image is pulled from the registry cache (or already warm).
  • Local developers can replicate CI exactly by running the same image with docker run.

Negative Consequences / Risks

  • Adding or upgrading a tool requires a rebuild and republish of the build image; this is a one-time friction per change rather than per job run.
  • The build-images repository must be maintained alongside the project.

Pros and Cons of the Options

Option 1 — Install tools at run time

  • Pro: No separate image maintenance.
  • Con: Non-reproducible — upstream package versions change silently.
  • Con: Slow — every job downloads packages.
  • Con: Fails when the network or upstream registry is unavailable.

Option 2 — Shared mutable builder VM

  • Pro: Tools installed once, not per job.
  • Con: State accumulates over time; hard to reproduce exact environment.
  • Con: Concurrent jobs may interfere with each other.

Option 3 — Pre-built Docker images

  • Pro: Hermetic, reproducible, fast.
  • Pro: Tool set is auditable via Dockerfile and image digest.
  • Con: Requires a separate build-images repository and CI pipeline.

Notes

Build images are distinct from runtime/service images. See the Containerization Guidelines for guidance on constructing minimal, secure runtime images.