Fix: Docker Build Cache Invalidated — Slow Builds on Every Run

Q: How do I fix "Docker Build Cache Invalidated — Slow Builds on Every Run"?

How to fix Docker layer cache being invalidated on every build — Dockerfile instruction order, COPY optimization, ARG vs ENV, BuildKit cache mounts, and .dockerignore.

The Problem

A Docker build that should use the cache rebuilds all layers on every run:

docker build -t myapp .

# Expected: Using cache for slow layers
Step 4/12 : RUN npm install
 ---> Using cache  ✓

# Actual: All layers rebuild every time
Step 4/12 : RUN npm install
 ---> Running in a1b2c3d4e5f6
added 1234 packages from 5678 contributors  # Full reinstall every build

Or the cache is invalidated by an unrelated file change:

COPY . .            ← Copies everything including README.md
RUN npm install     ← Invalidated every time any file changes, even non-package files

Or in CI/CD, the cache is never used because the build runs on a fresh runner:

# GitHub Actions — no cache between runs by default
Step 5/15: RUN pip install -r requirements.txt
Downloading flask-2.3.2-py3-none-any.whl  # Re-downloads every time

Why This Happens

Docker builds images layer by layer. Each instruction (RUN, COPY, ADD) creates a layer. Docker caches each layer and reuses it if:

The instruction string hasn’t changed
The parent layer hash is the same
For COPY/ADD: the file content hasn’t changed

Once one layer is invalidated, all subsequent layers are also invalidated — they can’t use cache because their parent layer changed. This makes instruction order critical.

The cascade effect is what makes Docker cache invalidation so frustrating. A single changed byte in a file touched by an early COPY instruction forces every layer below it to re-execute, even if those layers have nothing to do with the changed file. A 50-character edit to a README triggers a full npm install that takes three minutes.

Common cache-busting mistakes:

COPY . . before RUN npm install — any file change (even README.md) invalidates the COPY layer, causing npm install to re-run.
ARG BUILD_DATE or timestamps — dynamic values invalidate the layer on every build.
Not using .dockerignore — COPY . . includes node_modules, .git, and other artifacts that shouldn’t affect the cache but do.
apt-get update without pinned versions — package list updates invalidate the cache unexpectedly.
CI running on fresh machines — layer cache is per-machine unless explicitly saved/restored.
BuildKit vs legacy builder differences — BuildKit computes file checksums differently than the legacy builder. A file with identical content but a different modification timestamp may bust the cache on one builder but not the other.

Diagnostic Timeline

You suspect the Dockerfile instruction order is wrong, so you reorder COPY and RUN. But the cache still misses. This timeline shows how to pinpoint the exact layer that breaks.

Minute 0 — Run the build with full output. The default build output hides cache status. Use --progress=plain to see every layer:

DOCKER_BUILDKIT=1 docker build --progress=plain -t myapp . 2>&1 | tee build.log

Each layer shows either CACHED or the full execution output. Search the log for the first layer that is NOT cached — that is where the chain breaks.

Minute 2 — Identify the cache-busting layer. Suppose the output shows:

#6 [3/8] COPY package.json package-lock.json ./
#6 CACHED
#7 [4/8] RUN npm ci
#7 CACHED
#8 [5/8] COPY . .
#8 sha256:a1b2c3... 0.4s
#9 [6/8] RUN npm run build
#9 sha256:d4e5f6... 28.3s

Layer 8 (COPY . .) missed the cache. Every layer after it rebuilds. The question is: which file changed?

Minute 4 — Check what changed in the build context. Compare the build context between two builds. If .dockerignore is misconfigured, files like .git/ or node_modules/ are sent to the daemon and included in the COPY:

# See the build context size
docker build --no-cache -t test . 2>&1 | grep "Sending build context"
# "Sending build context to Docker daemon  245MB" ← too large

If the context is hundreds of megabytes, .dockerignore is missing or incomplete.

Minute 6 — Test with BuildKit’s cache debug. BuildKit exposes cache miss reasons with --export-cache diagnostics. For a quicker check, add --build-arg BUILDKIT_INLINE_CACHE=1 and compare the layer digests between two consecutive builds:

docker build -t myapp:v1 . 2>&1 | grep "sha256"
# Change nothing, build again:
docker build -t myapp:v2 . 2>&1 | grep "sha256"

If digests differ on COPY . . even though you changed no files, check for timestamp-based cache busting. Git operations (git checkout, git pull) update file modification times, causing Docker to see “changed” files even when their content is identical.

Minute 10 — Isolate the ARG problem. If your Dockerfile contains ARG instructions with dynamic values (ARG BUILD_DATE, ARG GIT_SHA), every build passes a different value. Move those ARGs below all expensive RUN layers, or switch to runtime ENV.

Fix 1: Order Instructions from Least to Most Frequently Changing

The most impactful change — put dependency installation before copying application code:

# WRONG — any file change busts the npm install cache
FROM node:20-alpine

WORKDIR /app

COPY . .                    # ← Everything copied first — changes every build
RUN npm ci                  # ← Reinstalls on every file change
RUN npm run build

# CORRECT — dependencies cached until package.json changes
FROM node:20-alpine

WORKDIR /app

# Step 1: Copy ONLY dependency files first (changes rarely)
COPY package.json package-lock.json ./

# Step 2: Install dependencies (uses cache if package files unchanged)
RUN npm ci

# Step 3: Copy application code (changes frequently — only invalidates layers below)
COPY . .

# Step 4: Build (depends on app code — runs when app code changes)
RUN npm run build

The principle: Instructions that change frequently go at the bottom. Anything above an unchanged instruction can be cached.

For Python projects:

# CORRECT order for Python
FROM python:3.12-slim

WORKDIR /app

COPY requirements.txt ./             # Only requirements file first
RUN pip install --no-cache-dir -r requirements.txt  # Cached until requirements.txt changes

COPY . .                             # App code (changes often — only invalidates CMD)
CMD ["uvicorn", "main:app", "--host", "0.0.0.0"]

For Go projects:

# CORRECT order for Go
FROM golang:1.22-alpine AS builder

WORKDIR /app

COPY go.mod go.sum ./               # Module files first
RUN go mod download                 # Cache modules — rarely changes

COPY . .                            # Source files
RUN go build -o server ./cmd/server

Fix 2: Create a Thorough .dockerignore

Without .dockerignore, COPY . . includes files that shouldn’t affect the image but change frequently, busting the cache:

# .dockerignore
# Version control
.git
.gitignore

# Dependencies (built inside Docker)
node_modules
vendor
__pycache__
*.pyc
*.pyo

# Build output
dist
build
.next
target

# Test files
.pytest_cache
coverage
.nyc_output
*.test.js

# Development files
.env
.env.local
*.env.development
docker-compose.override.yml

# Editor files
.vscode
.idea
*.swp
*.swo

# OS files
.DS_Store
Thumbs.db

# Documentation
*.md
docs

# CI/CD
.github
.circleci

# Logs
*.log
logs

Verify .dockerignore is working:

# Check what Docker sends to the build context (before any COPY instructions)
docker build --no-cache -t test . 2>&1 | head -5
# "Sending build context to Docker daemon  1.234MB"
# If this number is large (hundreds of MB), .dockerignore is insufficient

Fix 3: Use BuildKit Cache Mounts for Package Managers

Docker BuildKit’s --mount=type=cache persists package manager caches between builds without storing them in the image layer:

# Enable BuildKit (default in Docker 23.0+)
export DOCKER_BUILDKIT=1
# Or add to daemon.json: { "features": { "buildkit": true } }

# syntax=docker/dockerfile:1

FROM node:20-alpine

WORKDIR /app
COPY package.json package-lock.json ./

# Cache mount — npm cache persists between builds, not in the image
RUN --mount=type=cache,target=/root/.npm \
    npm ci

COPY . .
RUN --mount=type=cache,target=/root/.npm \
    npm run build

# Python with pip cache
FROM python:3.12-slim

WORKDIR /app
COPY requirements.txt ./

RUN --mount=type=cache,target=/root/.cache/pip \
    pip install -r requirements.txt

COPY . .

# Go with module cache
FROM golang:1.22

WORKDIR /app
COPY go.mod go.sum ./

RUN --mount=type=cache,target=/root/go/pkg/mod \
    go mod download

COPY . .
RUN --mount=type=cache,target=/root/go/pkg/mod \
    --mount=type=cache,target=/root/.cache/go-build \
    go build -o /app/server ./...

# apt-get with cache
FROM ubuntu:22.04

RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
    --mount=type=cache,target=/var/lib/apt,sharing=locked \
    apt-get update && apt-get install -y \
    curl \
    git \
    && rm -rf /var/lib/apt/lists/*

Cache mounts don’t affect the layer hash — they persist between builds on the same machine regardless of whether other layers changed.

Fix 4: Avoid Dynamic Values That Bust Cache

Variables or values that change every build invalidate all layers that depend on them:

# WRONG — timestamp changes every build, busts all subsequent layers
ARG BUILD_DATE
RUN echo "Built on $BUILD_DATE" >> /app/build-info.txt

# WRONG — git SHA from ARG causes rebuild every commit
ARG GIT_SHA
RUN echo $GIT_SHA > /app/version.txt

# CORRECT — use ARG AFTER all package installation layers
FROM node:20-alpine
WORKDIR /app

# All expensive layers first (no ARGs yet)
COPY package.json package-lock.json ./
RUN npm ci
COPY . .
RUN npm run build

# ARG/ENV near the end — only invalidates layers below this line
ARG GIT_SHA=unknown
ARG BUILD_DATE=unknown
RUN echo "{\"sha\": \"$GIT_SHA\", \"date\": \"$BUILD_DATE\"}" > /app/public/version.json

CMD ["node", "server.js"]

Pass dynamic values at runtime instead of build time:

# Better: use environment variables at runtime
ENV APP_VERSION=unknown

# Pass at runtime:
# docker run -e APP_VERSION=$(git rev-parse HEAD) myapp

Fix 5: Pin apt-get Packages to Prevent Cache Invalidation

apt-get update refreshes the package list — the result changes over time, busting the cache:

# WRONG — apt-get update changes output over time
RUN apt-get update
RUN apt-get install -y curl   # ← These are separate layers AND update may re-run

# STILL WRONG — separate layers, and update may return different results later
RUN apt-get update && apt-get install -y curl

# CORRECT — combined into one layer, pin versions, clean up in same layer
RUN apt-get update && apt-get install -y --no-install-recommends \
    curl=7.88.1-10+deb12u5 \  # Pin version
    git=1:2.39.2-1.1 \
    && rm -rf /var/lib/apt/lists/*   # Remove cache from layer — reduces image size

For development where pinning is inconvenient, use a base image that already has tools:

# Use an image that already has curl and git — no apt-get needed
FROM node:20-alpine  # alpine includes apk (faster) and common tools

Fix 6: Cache Docker Layers in CI/CD

On CI/CD, each run starts with a fresh environment — no local layer cache. Explicit cache management is required:

GitHub Actions with docker/build-push-action:

# .github/workflows/build.yml
- name: Set up Docker Buildx
  uses: docker/setup-buildx-action@v3

- name: Build and push
  uses: docker/build-push-action@v5
  with:
    context: .
    push: true
    tags: myapp:latest
    cache-from: type=gha          # Pull cache from GitHub Actions Cache
    cache-to: type=gha,mode=max   # Push cache to GitHub Actions Cache
    # mode=max caches all layers, not just the final stage

Registry-based cache (works across CI systems):

- name: Build with registry cache
  uses: docker/build-push-action@v5
  with:
    context: .
    cache-from: type=registry,ref=ghcr.io/myorg/myapp:buildcache
    cache-to: type=registry,ref=ghcr.io/myorg/myapp:buildcache,mode=max
    push: true
    tags: ghcr.io/myorg/myapp:latest

Inline cache (simpler but less effective):

- name: Build
  run: |
    docker pull myapp:latest || true   # Pull previous image for cache
    docker build \
      --cache-from myapp:latest \
      --build-arg BUILDKIT_INLINE_CACHE=1 \
      -t myapp:latest .
    docker push myapp:latest

Fix 7: Use Multi-Stage Builds to Minimize Cache Impact

Multi-stage builds keep build tools out of the final image and allow independent caching of each stage:

# syntax=docker/dockerfile:1

# Stage 1: Dependencies (cached unless package.json changes)
FROM node:20-alpine AS deps
WORKDIR /app
COPY package.json package-lock.json ./
RUN --mount=type=cache,target=/root/.npm npm ci

# Stage 2: Builder (cached unless source code changes)
FROM node:20-alpine AS builder
WORKDIR /app
COPY --from=deps /app/node_modules ./node_modules
COPY . .
RUN npm run build

# Stage 3: Production (minimal image — only runtime files)
FROM node:20-alpine AS runner
WORKDIR /app
ENV NODE_ENV=production

COPY --from=builder /app/dist ./dist
COPY --from=builder /app/node_modules ./node_modules

EXPOSE 3000
CMD ["node", "dist/server.js"]

Each stage is cached independently — a source code change only rebuilds the builder and runner stages, not the deps stage.

Still Not Working?

Force a full rebuild — to verify cache is working, first confirm a clean build works correctly, then make a change and verify cache is used:

# Force no-cache rebuild
docker build --no-cache -t myapp .

# Next run — should use cache for unchanged layers
docker build -t myapp .

BuildKit not enabled — classic Docker builder (without BuildKit) has less effective caching. Enable BuildKit:

DOCKER_BUILDKIT=1 docker build -t myapp .

# Or set permanently in /etc/docker/daemon.json:
{ "features": { "buildkit": true } }

Image from different registry not compatible for cache — --cache-from requires the image to have been built with --build-arg BUILDKIT_INLINE_CACHE=1 or with the type=registry cache exporter to be useful as a cache source.

.dockerignore not in the right directory — .dockerignore must be in the same directory as the Dockerfile (the build context root), not in subdirectories.

COPY with --chown or --chmod changes the layer hash — even if the file content is identical, adding or changing --chown=node:node on a COPY instruction produces a different layer hash. If you recently added ownership flags, every layer from that point down rebuilds once, then caches normally on subsequent builds.

Git checkout changes file timestamps — git checkout and git pull update the modification time of every file they touch. Docker’s legacy builder uses mtime as part of its cache key for COPY. If you build, then git pull (even with no changes to your branch), the modified timestamps on unchanged files may differ, busting the cache. BuildKit uses file content hashes instead of mtime, which avoids this problem. Switch to BuildKit if you see phantom cache misses after git operations.

ADD with a remote URL never caches — ADD https://example.com/file.tar.gz /app/ fetches the URL on every build because Docker cannot determine whether the remote content changed. Use RUN curl -o combined with a cache mount instead, or download the file in a previous CI step and COPY it.