Fix: Docker Secrets Not Working — BuildKit --secret Not Mounting, Compose Secrets Undefined, or Secret Leaking into Image

The Problem

BuildKit secret mounts don’t appear inside the container during build:

# Dockerfile
RUN --mount=type=secret,id=npm_token \
    cat /run/secrets/npm_token  # Empty or file not found

docker build --secret id=npm_token,src=.npmrc .
# Error: secret file not found: /run/secrets/npm_token

Or Docker Compose secrets aren’t accessible in the container:

# docker-compose.yml
services:
  app:
    secrets:
      - db_password

secrets:
  db_password:
    file: ./secrets/db_password.txt

docker compose up
# cat /run/secrets/db_password  # Empty or permission denied

Or a secret was used during build and ended up in the image layers:

docker history myimage
# IMAGE          CREATED_AT     CREATED_BY
# abc123         2 min ago      RUN npm install  # Secret was ARG — leaked!

Why This Happens

Docker has two separate secret mechanisms that are often confused, and the names overlap enough that even experienced engineers reach for the wrong one. The first instinct when you see “secret undefined” is to mount it harder — add another secrets: block, copy the file to a new path — but the real fix is usually picking the right mechanism for the phase you’re in.

A build-time secret and a runtime secret are not the same thing despite both mounting at /run/secrets/. A docker-compose.yml secrets block does nothing during docker build. A --secret flag passed to docker build does nothing at container runtime. And the syntax that ties them together (# syntax=docker/dockerfile:1, BuildKit enabled, swarm mode vs compose mode) silently degrades when any one piece is missing. That silence is why secret bugs feel like a maze.

• Build-time secrets (BuildKit --mount=type=secret) — secrets mounted during docker build. They’re available only in the specific RUN instruction that mounts them, never baked into image layers. Requires BuildKit to be enabled.
• Runtime secrets (Docker Compose / Swarm secrets) — secrets mounted into the container filesystem at /run/secrets/<name> when the container runs. These are a runtime mechanism — they don’t exist during the build phase.
• ARG and ENV are not secrets — any value passed as ARG or ENV is stored in the image layer history and visible with docker history. Never use them for credentials.
• BuildKit not enabled — --mount=type=secret requires BuildKit. Without it, the --mount syntax is silently ignored or causes a build error.

Diagnostic Timeline

The path from “secret not found” to a working build usually looks like this:

Minute 0 — first guess: mount the secret. You add a secrets: block to docker-compose.yml pointing at ./secrets/db_password.txt, rebuild the image, and the RUN instruction still can’t read /run/secrets/db_password. The instinct is wrong: compose secrets are runtime-only and never appear during docker build. The container will see them, but the build phase never will.

Minute 4 — check whether BuildKit is enabled. Run docker build --progress=plain . and look at the first line of output. If you see “Building with classic builder” or the build skips the # syntax=... line, BuildKit is off. Export DOCKER_BUILDKIT=1 or switch to docker buildx build. Without BuildKit, --mount=type=secret is silently ignored on older Docker versions and outright rejected on newer ones.

Minute 8 — confirm the # syntax=docker/dockerfile:1 directive. Even with BuildKit enabled, the secret mount syntax requires the Dockerfile frontend. Missing the syntax line at the top causes --mount=type=secret to parse as a plain unknown flag in some Docker versions, leaving you with an empty secret path. Add the line as the very first line of the Dockerfile.

Minute 12 — distinguish swarm secrets from compose secrets. If you’re running docker stack deploy, secrets must be created with docker secret create first and referenced as external: true in the compose file. If you’re running docker compose up, the file: or environment: form is read directly. Mixing the two — declaring an external secret without creating it, or declaring a file secret on swarm — silently mounts an empty file.

Minute 18 — verify the build-time secret reached the container. Add RUN --mount=type=secret,id=npm_token ls -la /run/secrets/ to your Dockerfile temporarily. If the file is there but empty, the host path in --secret id=npm_token,src=... is wrong. If the directory itself is missing, BuildKit isn’t being used. If the file is there and has content but your script can’t read it, you’re likely running the read step in a different RUN block — secrets are scoped to a single instruction.

Fix 1: Use BuildKit Secret Mounts for Build-Time Credentials

BuildKit secret mounts let you access credentials during build without baking them into the image:

# Enable BuildKit (required)
export DOCKER_BUILDKIT=1

# Or use the docker buildx command (always uses BuildKit)
docker buildx build .

# syntax=docker/dockerfile:1
FROM node:20-alpine

WORKDIR /app
COPY package*.json ./

# Mount the secret for this RUN only — never stored in layers
RUN --mount=type=secret,id=npm_token \
    NPM_TOKEN=$(cat /run/secrets/npm_token) \
    npm config set //registry.npmjs.org/:_authToken=$NPM_TOKEN && \
    npm ci && \
    npm config delete //registry.npmjs.org/:_authToken

COPY . .
RUN npm run build

# Pass the secret from a file
docker build --secret id=npm_token,src=$HOME/.npmrc .

# Or from an environment variable
docker build --secret id=npm_token,env=NPM_TOKEN .

# Or using buildx
docker buildx build --secret id=npm_token,src=$HOME/.npmrc .

Multiple secrets:

# syntax=docker/dockerfile:1
FROM python:3.12-slim

WORKDIR /app
COPY requirements.txt .

RUN --mount=type=secret,id=pip_config \
    pip install \
        --config-file /run/secrets/pip_config \
        -r requirements.txt

docker build \
    --secret id=pip_config,src=./pip.conf \
    --secret id=ca_cert,src=./ca.crt \
    .

Important: The # syntax=docker/dockerfile:1 comment at the top of your Dockerfile enables the BuildKit frontend. Without it, --mount=type=secret may not be recognized even when BuildKit is enabled.

Fix 2: Fix Docker Compose Runtime Secrets

Docker Compose secrets are injected at container runtime, not build time:

# docker-compose.yml
version: "3.8"

services:
  app:
    image: myapp:latest
    secrets:
      - db_password
      - api_key
    environment:
      # Reference secret path in app config, not the secret itself
      DB_PASSWORD_FILE: /run/secrets/db_password
    ports:
      - "3000:3000"

  db:
    image: postgres:16
    secrets:
      - db_password
    environment:
      POSTGRES_PASSWORD_FILE: /run/secrets/db_password
      POSTGRES_USER: myapp

secrets:
  db_password:
    file: ./secrets/db_password.txt  # Plain text file with the secret value

  api_key:
    environment: API_KEY  # Read from host environment variable

# Create the secrets directory (never commit this)
mkdir -p secrets
echo -n "mysecretpassword" > secrets/db_password.txt

# Add to .gitignore
echo "secrets/" >> .gitignore

docker compose up

Access secrets inside the container:

# Secrets are mounted at /run/secrets/<secret-name>
docker exec myapp cat /run/secrets/db_password

# In your application code

# Python — read secret from file
import os

def get_secret(name: str) -> str:
    secret_path = f"/run/secrets/{name}"
    if os.path.exists(secret_path):
        with open(secret_path) as f:
            return f.read().strip()
    # Fallback to environment variable for local dev
    return os.environ.get(name.upper(), "")

db_password = get_secret("db_password")

// Node.js — read secret from file
import { readFileSync, existsSync } from "fs";

function getSecret(name) {
  const secretPath = `/run/secrets/${name}`;
  if (existsSync(secretPath)) {
    return readFileSync(secretPath, "utf8").trim();
  }
  return process.env[name.toUpperCase()] ?? "";
}

const dbPassword = getSecret("db_password");

Docker Swarm secrets (production):

# Create a secret in Swarm
echo "mysecretpassword" | docker secret create db_password -

# Or from a file
docker secret create db_password ./secrets/db_password.txt

# List secrets (values are never shown)
docker secret ls

# Use in a service
docker service create \
    --name myapp \
    --secret db_password \
    myapp:latest

Fix 3: Verify Secrets Don’t Leak into Image Layers

After building, check that credentials aren’t stored in the image:

# Check image history for sensitive values
docker history --no-trunc myimage | grep -i "token\|password\|secret\|key"

# Inspect all layers
docker save myimage | tar -tv | grep -v "^d"

# Scan with dive (interactive layer inspector)
docker run --rm -it \
    -v /var/run/docker.sock:/var/run/docker.sock \
    wagoodman/dive myimage

Common mistakes that leak secrets:

# WRONG — ARG values are visible in docker history
ARG NPM_TOKEN
RUN npm config set //registry.npmjs.org/:_authToken=$NPM_TOKEN && npm ci

# WRONG — ENV persists in the image
ENV NPM_TOKEN=mytoken
RUN npm ci

# WRONG — copying a file with secrets into the image
COPY .npmrc .
RUN npm ci
# The .npmrc is now in the layer even if you RUN rm .npmrc later

# CORRECT — BuildKit secret mount (never stored)
RUN --mount=type=secret,id=npm_token \
    NPM_TOKEN=$(cat /run/secrets/npm_token) \
    npm config set //registry.npmjs.org/:_authToken=$NPM_TOKEN && \
    npm ci && \
    npm config delete //registry.npmjs.org/:_authToken

Multi-stage builds for isolation:

# syntax=docker/dockerfile:1

# Build stage — has access to secrets
FROM node:20-alpine AS builder
WORKDIR /app
COPY package*.json ./

RUN --mount=type=secret,id=npm_token \
    NPM_TOKEN=$(cat /run/secrets/npm_token) \
    npm config set //registry.npmjs.org/:_authToken=$NPM_TOKEN && \
    npm ci

COPY . .
RUN npm run build

# Production stage — clean image, no secrets
FROM node:20-alpine AS production
WORKDIR /app
COPY --from=builder /app/dist ./dist
COPY --from=builder /app/node_modules ./node_modules
# Secrets never touch this stage
EXPOSE 3000
CMD ["node", "dist/index.js"]

Fix 4: Integrate Secrets in CI/CD Pipelines

Never store secrets in Dockerfiles or docker-compose files committed to version control:

GitHub Actions:

# .github/workflows/build.yml
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3

      - name: Build with secret
        run: |
          docker buildx build \
            --secret id=npm_token,env=NPM_TOKEN \
            --tag myimage:latest \
            --push \
            .
        env:
          NPM_TOKEN: ${{ secrets.NPM_TOKEN }}
          # NPM_TOKEN is set in GitHub repo Settings → Secrets

GitLab CI:

# .gitlab-ci.yml
build:
  stage: build
  script:
    - docker buildx build
        --secret id=npm_token,env=NPM_TOKEN
        --tag $CI_REGISTRY_IMAGE:latest
        --push
        .
  variables:
    DOCKER_BUILDKIT: "1"
  # NPM_TOKEN set in GitLab CI/CD Settings → Variables

Using Docker config for registry auth (not a build-time secret):

# Log in to a private registry — stores credentials in ~/.docker/config.json
docker login registry.example.com -u myuser -p mypassword

# Build and push — registry auth is handled by Docker daemon, not build secrets
docker buildx build --push --tag registry.example.com/myimage:latest .

Fix 5: Use External Secret Managers

For production, pull secrets from a secret manager rather than files or environment variables:

AWS Secrets Manager:

# syntax=docker/dockerfile:1
FROM python:3.12-slim

# Install AWS CLI or SDK at build time
RUN pip install boto3

COPY entrypoint.sh /entrypoint.sh
COPY app.py /app.py

# Fetch secrets at runtime in the entrypoint
ENTRYPOINT ["/entrypoint.sh"]

#!/bin/sh
# entrypoint.sh — fetch secrets at container start
DB_PASSWORD=$(aws secretsmanager get-secret-value \
    --secret-id prod/myapp/db_password \
    --query SecretString \
    --output text)
export DB_PASSWORD

exec python app.py

HashiCorp Vault with Docker:

# docker-compose.yml with Vault Agent sidecar
services:
  vault-agent:
    image: hashicorp/vault:latest
    command: agent -config=/vault/config/agent.hcl
    volumes:
      - ./vault-agent.hcl:/vault/config/agent.hcl
      - secrets-volume:/secrets

  app:
    image: myapp:latest
    volumes:
      - secrets-volume:/run/secrets:ro
    depends_on:
      - vault-agent

volumes:
  secrets-volume:

Kubernetes Secrets (for K8s deployments):

# k8s-secret.yaml — create secret
apiVersion: v1
kind: Secret
metadata:
  name: myapp-secrets
type: Opaque
data:
  db_password: bXlzZWNyZXRwYXNzd29yZA==  # base64 encoded
  api_key: c29tZWFwaWtleQ==

---
# deployment.yaml — mount as files or env vars
apiVersion: apps/v1
kind: Deployment
spec:
  template:
    spec:
      containers:
        - name: myapp
          image: myapp:latest
          # Mount as files (more secure)
          volumeMounts:
            - name: secrets
              mountPath: /run/secrets
              readOnly: true
          # Or inject as environment variables
          env:
            - name: DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: myapp-secrets
                  key: db_password
      volumes:
        - name: secrets
          secret:
            secretName: myapp-secrets

Fix 6: Handle .env Files and Local Development

For local development, use .env files with Docker Compose — but never commit them:

# .env — local development only, never commit this
DB_PASSWORD=localdevpassword
API_KEY=devkey123
NPM_TOKEN=mytoken

# docker-compose.yml — pass env vars to container
services:
  app:
    image: myapp:latest
    env_file:
      - .env  # Docker Compose reads this automatically
    # Or explicitly:
    environment:
      - DB_PASSWORD=${DB_PASSWORD}
      - API_KEY=${API_KEY}

# .gitignore — always exclude secret files
.env
.env.local
.env.production
secrets/
*.pem
*.key

Template for shared development setup:

# .env.example — commit this as a template
DB_PASSWORD=changeme
API_KEY=your_api_key_here
NPM_TOKEN=your_npm_token_here

# README instructions for new developers:
# cp .env.example .env
# Fill in real values in .env

Validate secrets are present before starting:

#!/bin/sh
# docker-entrypoint.sh — fail early if required secrets are missing

required_secrets="db_password api_key"

for secret in $required_secrets; do
    if [ ! -f "/run/secrets/$secret" ] && [ -z "$(eval echo \$$secret)" ]; then
        echo "ERROR: Required secret '$secret' is not set"
        echo "  Mount it as a Docker secret at /run/secrets/$secret"
        echo "  Or set the environment variable $secret"
        exit 1
    fi
done

exec "$@"

Still Not Working?

/run/secrets/ directory doesn’t exist in the container — this directory is created by Docker when secrets are mounted. If your container doesn’t use Docker Compose secrets or Swarm secrets, the directory won’t be there. For standalone containers (docker run), mount secrets manually with -v or use environment variables. The /run/secrets/ convention is a Docker feature, not a system default.

BuildKit secret mount path — default is /run/secrets/<id> — if your RUN command looks for the secret in a different location, it won’t find it. The default mount point is always /run/secrets/<id> where <id> is the --secret id=<id> value. Override with target=:

RUN --mount=type=secret,id=npm_token,target=/root/.npmrc \
    npm ci
# Secret is now at /root/.npmrc instead of /run/secrets/npm_token

docker compose up uses old secrets after changes — Docker Compose caches container state. If you changed the secret file, recreate the containers:

docker compose down
docker compose up --force-recreate

Permission denied on /run/secrets/<name> — secrets are owned by root with mode 0400 by default. If your process runs as a non-root user, either change the secret permissions in your Compose config or read it as root in the entrypoint and export as an environment variable:

secrets:
  db_password:
    file: ./secrets/db_password.txt
    # Docker Swarm supports uid/gid/mode settings:
    # mode: 0440
    # uid: "1000"

Secret value contains a trailing newline that breaks comparisons — echo "password" > secrets/db_password.txt writes password\n, and many services (Postgres, Redis auth, JWT signing) include the newline in the actual password. Use echo -n or printf "password" to omit the trailing newline. To verify what’s actually stored, run xxd secrets/db_password.txt | head and check for 0a at the end. The same applies to secrets fetched from cloud secret managers — strip trailing whitespace before comparison.

Compose secrets work in up but disappear in run — docker compose run <service> creates a one-off container that may not mount secrets unless --use-aliases and --service-ports defaults are explicit, and certain Compose versions skip the secrets: block for run entirely. Use docker compose up <service> for anything that needs secrets, or pin to a Compose version where run honors them (v2.20+).

Build secret cached across builds — BuildKit caches the result of RUN --mount=type=secret based on the command and Dockerfile contents, not the secret value. If you rotate the secret but the Dockerfile didn’t change, BuildKit reuses the cached layer with the old secret embedded in whatever output it produced (a fetched dependency, a generated config). Force a rebuild with --no-cache on the affected stages, or add a cache-busting ARG whose value changes when the secret rotates.

For related Docker issues, see Fix: Docker Build ARG Not Available, Fix: Docker Compose Healthcheck Not Working, Fix: Docker Healthcheck Failing, and Fix: Docker Compose Env Not Loaded.