Every second of CI/CD build time multiplies across your team. If 5 developers each push 10 commits per day and your Docker build takes 5 minutes, that's 250 minutes of developer waiting time per day — plus CI compute costs.
Understanding Docker's layer cache can reduce builds from 5 minutes to 15 seconds.
How Layers Work: The Mental Model
Every instruction in a Dockerfile creates a layer — an immutable, content-addressed filesystem snapshot stored by its SHA256 hash.
When you build an image, Docker checks each instruction against its cache:
- Is this instruction identical to the previous build?
- Is the content being copied identical (for
COPYinstructions)? - If yes to both → cache hit (reuse the layer, skip re-execution)
- If no → cache miss (re-execute) AND invalidate all subsequent layers
The golden rule: A cache miss at layer N forces all layers N+1, N+2, N+3... to also re-execute, even if they haven't changed.
The Classic Mistake: Copy Before Install
Problem: Every time you change a single line in server.js, the COPY . . layer detects a change. This invalidates the RUN npm install layer below it. Docker re-downloads and re-installs all node_modules — even though package.json hasn't changed.
On a project with 500 npm dependencies, npm install can take 3–5 minutes. Doing it on every commit is expensive.
The Fix: Layer Ordering Principle
Order your instructions from "least likely to change" → "most likely to change":
Now when you change server.js:
- Layer 1 (
FROM): cache hit ✅ - Layer 2 (
WORKDIR): cache hit ✅ - Layer 3 (
COPY package.json): cache hit ✅ (package.json didn't change) - Layer 4 (
RUN npm ci): cache hit ✅ (skipped entirely!) - Layer 5 (
COPY . .): cache miss ❌ (only this layer reruns — takes 0.1s)
Result: Build time drops from 5 minutes → 3 seconds.
Language-Specific Cache Optimization Patterns
Python (pip)
Go (go modules)
Java (Maven)
Build Cache on CI/CD: The Challenge
The per-build cache that speeds up local development is lost between CI runs — each GitHub Actions job starts with a fresh runner.
Without cache, every CI build re-downloads all dependencies. With cache strategies, you restore the cached layers before building.
Strategy 1: GitHub Actions Cache (BuildKit)
mode=max caches every intermediate layer, not just the final image. This maximises cache reuse across branches.
Strategy 2: Registry Cache (cache layers in the container registry)
The cache layers are stored as a special manifest in your registry — works across different CI runners and machines.
Strategy 3: Local Registry Cache (self-hosted runners)
Reducing Layer Count and Image Size
Combine RUN Commands
Use --no-cache Flags for Package Managers
Clean Up in the Same RUN Layer
The --virtual flag groups the packages under a virtual name, making bulk removal easy. The total image does NOT contain the build tools because the install and delete happen in the same layer.
Analysing Image Size and Layers
dive shows you exactly what files each layer adds or removes — invaluable for finding what's bloating your image.
BuildKit: Docker's Modern Build Engine
BuildKit is the default build engine since Docker 23. It provides:
- Parallel stage execution in multi-stage builds
- Better cache management
- Secret mounts (pass secrets at build time without leaking into layers)
- SSH agent forwarding (for private git repos)
Summary
Layer caching is one of the highest-impact optimizations available:
- Order instructions from least-to-most frequently changing —
COPY package.jsonbeforeCOPY . . - A cache miss at any layer invalidates all subsequent layers — even unchanged ones
- Combine related RUN commands into a single instruction; clean up (package caches, temp files) in the same
RUN - In CI/CD, use
type=ghaortype=registrycache backends to persist layer cache between runs diveis the best tool for inspecting what's inside each image layer- BuildKit (
--mount=type=secret) allows secure secret injection at build time without layer leakage
In the next lesson, you will learn multi-stage builds — separating compile-time environments from runtime environments to produce minimal production images.