Fix: AWS Lambda Layer Not Working — Module Not Found or Layer Not Applied

The Problem

A Lambda function using a Layer throws a module not found error:

Error: Cannot find module 'sharp'
Runtime.ImportModuleError: Unable to import module 'lambda_function': No module named 'pandas'

Despite the Layer being attached in the console, the module isn’t available.

Or the Layer is attached but Lambda uses the wrong version:

Error: Runtime.ImportModuleError: Unable to import module 'handler':
  /opt/python/lib/python3.9/site-packages/numpy/core/_multiarray_umath.cpython-39...
  ...incompatible with python3.11

Or a Layer created for one architecture (x86) fails on another (arm64):

Error: /opt/nodejs/node_modules/sharp/build/Release/sharp-linux-x64.node:
  invalid ELF header

Why This Happens

Lambda Layers have strict requirements that are easy to get wrong, and most failures fall into one of a few categories that all surface as the same Cannot find module or Unable to import module error at cold start.

The first category is layout. Lambda mounts every attached layer at /opt and only searches a handful of paths inside that mount for code. For Node.js it looks at /opt/nodejs/node_modules/. For Python it looks at /opt/python/ first and then /opt/python/lib/python<X.Y>/site-packages/. For Ruby it looks at /opt/ruby/gems/<version>/. For Java it looks at /opt/java/lib/. If your zip places files anywhere else — for instance at the zip root, or inside a dist/ folder — Lambda will not find them. The fact that the layer attached successfully tells you nothing about whether the runtime will discover the code inside.

The second category is binary compatibility. Lambda runs on Amazon Linux 2 (and Amazon Linux 2023 for newer runtimes), and native extensions like numpy, pandas, psycopg2, cryptography, sharp, and bcrypt ship compiled binaries that are pinned to a specific CPU architecture, a specific OS family, and often a specific Python or Node minor version. A wheel built on macOS or installed by a developer with pip install numpy on their laptop will not load on Lambda. The error usually appears as invalid ELF header, cannot open shared object file, or a numpy _multiarray_umath import failure.

The third category is the runtime/architecture mismatch between the layer and the function. A layer published with CompatibleRuntimes: [python3.11] will not attach to a python3.12 function via the console, and even when it does attach via CLI, the C extensions inside it will refuse to load. Likewise, an x86_64 layer cannot be used by an arm64 (Graviton2) function — the binaries simply will not execute.

Finally, layer order and merge semantics confuse a lot of teams. Lambda extracts every attached layer into /opt in the order they appear on the function configuration, and later layers overwrite earlier ones when paths collide. If two layers both ship pandas, you get whichever one was attached last. This is why a “working” layer can stop working when an SRE adds a second layer above it.

Version History: Lambda Layers and Packaging Options

Lambda layers and the surrounding packaging story have evolved enough that “what to use” depends on when your function was originally built.

• Lambda layers GA (November 2018) — re:Invent 2018 introduced layers as a way to share code and dependencies across functions. The original limits (still in force) are 5 layers per function and 250 MB total unzipped across the function code and all layers.
• Lambda Extensions (October 2020) — layers gained the ability to ship sidecar processes via the Extensions API, used today for observability agents (Datadog, New Relic), secrets caching, and the official AWS Parameters and Secrets Lambda Extension.
• Container image support (December 2020) — re:Invent 2020 added container packaging with a 10 GB image limit. This is the only way to ship dependencies larger than 250 MB (PyTorch, Playwright, headless Chrome). Container images replace layers for that use case rather than supplementing them.
• arm64 / Graviton2 support (September 2021) — Lambda gained arm64 runtimes that are roughly 20% cheaper and ~20% faster for many workloads. Layers became architecture-specific: a layer built for x86_64 will not run on arm64 and vice versa.
• Runtime deprecations (ongoing) — Node.js 12, Python 3.7, and earlier runtimes have been retired. When a runtime is deprecated, you must rebuild layers against a supported runtime; the old binaries will not load on the replacement.
• Amazon Linux 2023 base (2023+) — newer runtimes such as Node.js 20 and Python 3.12 ship on AL2023 instead of AL2. C extensions built against AL2’s glibc may fail on AL2023 with GLIBC_2.34 not found. Rebuild on the matching base image (public.ecr.aws/lambda/python:3.12).
• SnapStart for Python and Node.js (2024) — extended SnapStart beyond Java. Layers with heavy initialization code (boto3, sqlalchemy) became cheaper to use because the cold-start cost is paid once at snapshot time.

A practical rule: if your dependencies fit comfortably under 50 MB zipped and you don’t need OS-level tooling, layers are still the right answer. If you are reaching for --platform manylinux2014_x86_64 and a build container anyway, container images are usually less fragile.

Fix 1: Use the Correct Directory Structure

The directory structure inside the layer zip is mandatory:

Node.js layer zip structure:
nodejs/
  node_modules/
    sharp/
    axios/
    ...

Python layer zip structure:
python/
  lib/
    python3.11/
      site-packages/
        pandas/
        numpy/
        ...
  OR (shorter path, also works):
python/
  pandas/
  numpy/
  ...

Ruby layer:
ruby/
  gems/
    3.2.0/
      gems/
        ...

Java layer:
java/
  lib/
    my-library.jar

Build a Node.js Layer correctly:

# Create the correct directory structure
mkdir -p layer/nodejs

# Install packages INTO the layer directory
cd layer/nodejs
npm init -y
npm install sharp axios lodash

# Go back and zip the layer directory
cd ../..
zip -r layer.zip layer/nodejs/node_modules

# WRONG — this would zip the top-level (missing nodejs/ prefix)
# zip -r layer.zip node_modules/

Build a Python Layer correctly:

# Method 1: Install directly to the layer path
mkdir -p layer/python
pip install pandas numpy -t layer/python/

# Zip preserving the 'python/' directory
cd layer
zip -r ../layer.zip python/

# Method 2: Using pip with --target
pip install \
  --platform manylinux2014_x86_64 \
  --target=layer/python \
  --implementation cp \
  --python-version 3.11 \
  --only-binary=:all: \
  pandas numpy

zip -r layer.zip python/

Fix 2: Build for the Correct Architecture

Native packages (numpy, pandas, psycopg2, sharp) must be compiled for the Lambda runtime OS and architecture:

# Build for Linux x86_64 (default Lambda architecture)
# Use Docker to match Lambda's OS environment

# Node.js — build inside the Lambda Docker image
docker run --rm \
  -v $(pwd):/workspace \
  -w /workspace/layer/nodejs \
  public.ecr.aws/lambda/nodejs:20 \
  npm install sharp

# Python — use pip's --platform flag for cross-compilation
pip install \
  --platform manylinux2014_x86_64 \
  --target=./layer/python \
  --implementation cp \
  --python-version 311 \
  --only-binary=:all: \
  pandas numpy psycopg2-binary

# For arm64 (Graviton Lambda functions)
pip install \
  --platform manylinux2014_aarch64 \
  --target=./layer/python \
  --implementation cp \
  --python-version 311 \
  --only-binary=:all: \
  pandas numpy

Using AWS SAM or CDK to build layers:

# template.yaml (SAM)
Resources:
  DependenciesLayer:
    Type: AWS::Serverless::LayerVersion
    Properties:
      ContentUri: layer/
      CompatibleRuntimes:
        - python3.11
      CompatibleArchitectures:
        - x86_64
    Metadata:
      BuildMethod: python3.11  # SAM builds this layer in a Lambda-compatible environment

# SAM builds the layer with the correct environment
sam build
sam deploy

Fix 3: Configure the Layer in Your Function

Attaching a layer in the console vs code vs CLI:

# AWS CLI — attach layer to existing function
aws lambda update-function-configuration \
  --function-name my-function \
  --layers arn:aws:lambda:us-east-1:123456789:layer:my-layer:3

# Attach multiple layers (order matters — later layers override earlier)
aws lambda update-function-configuration \
  --function-name my-function \
  --layers \
    arn:aws:lambda:us-east-1:123456789:layer:base-layer:1 \
    arn:aws:lambda:us-east-1:123456789:layer:my-overrides:2

Terraform:

resource "aws_lambda_function" "my_function" {
  function_name = "my-function"
  handler       = "index.handler"
  runtime       = "nodejs20.x"
  filename      = "function.zip"
  role          = aws_iam_role.lambda_role.arn

  layers = [
    aws_lambda_layer_version.my_layer.arn,
  ]

  architectures = ["x86_64"]  # Must match layer architecture
}

resource "aws_lambda_layer_version" "my_layer" {
  filename            = "layer.zip"
  layer_name          = "my-dependencies"
  compatible_runtimes = ["nodejs20.x"]
  compatible_architectures = ["x86_64"]
}

CDK:

import * as lambda from 'aws-cdk-lib/aws-lambda';

const dependenciesLayer = new lambda.LayerVersion(this, 'DependenciesLayer', {
  code: lambda.Code.fromAsset('layer'),
  compatibleRuntimes: [lambda.Runtime.NODEJS_20_X],
  compatibleArchitectures: [lambda.Architecture.X86_64],
  description: 'Node.js dependencies',
});

const myFunction = new lambda.Function(this, 'MyFunction', {
  runtime: lambda.Runtime.NODEJS_20_X,
  architecture: lambda.Architecture.X86_64,
  handler: 'index.handler',
  code: lambda.Code.fromAsset('src'),
  layers: [dependenciesLayer],
});

Fix 4: Use Public Layers for Common Libraries

AWS and the community publish ready-made layers for common libraries:

# AWS SDK for JavaScript (Node.js 16 and below only — Node 18+ includes it)
# Check available public layers:
aws lambda list-layers --compatible-runtime nodejs18.x

# Klayers (community Python layers by Keith Rozario)
# Pre-built for each Python version and region
# https://github.com/keithrozario/Klayers

# Get the ARN for pandas in us-east-1, Python 3.11
aws lambda list-layer-versions \
  --layer-name arn:aws:lambda:us-east-1:770693421928:layer:Klayers-p311-pandas \
  --query 'LayerVersions[0].LayerVersionArn'

# Use in your function
aws lambda update-function-configuration \
  --function-name my-function \
  --layers arn:aws:lambda:us-east-1:770693421928:layer:Klayers-p311-pandas:13

AWS Parameters and Secrets Lambda Extension (official layer):

# Add the Parameters and Secrets Extension layer
aws lambda update-function-configuration \
  --function-name my-function \
  --layers arn:aws:lambda:us-east-1:177933569100:layer:AWS-Parameters-and-Secrets-Lambda-Extension:11

Fix 5: Debug Layer Contents

Verify the layer is mounted correctly inside the Lambda environment:

# Python — debug handler to inspect the layer
import os
import sys

def handler(event, context):
    # List /opt directory (where layers are mounted)
    opt_contents = os.listdir('/opt')
    print("Contents of /opt:", opt_contents)

    # Check if python packages are available
    python_path = '/opt/python'
    if os.path.exists(python_path):
        print("Python layer contents:", os.listdir(python_path))

    # Check sys.path to see where Python looks for modules
    print("sys.path:", sys.path)

    # Try importing the module
    try:
        import pandas
        print("pandas version:", pandas.__version__)
    except ImportError as e:
        print("Import error:", e)

    return {"statusCode": 200}

// Node.js — debug handler
exports.handler = async (event) => {
  const fs = require('fs');
  const path = require('path');

  // /opt is where layers are mounted
  const optContents = fs.readdirSync('/opt');
  console.log('Contents of /opt:', optContents);

  // Node.js layer path
  const nodeModulesPath = '/opt/nodejs/node_modules';
  if (fs.existsSync(nodeModulesPath)) {
    console.log('Layer modules:', fs.readdirSync(nodeModulesPath));
  }

  // Try requiring the module
  try {
    const sharp = require('sharp');
    console.log('sharp version:', sharp.versions);
  } catch (e) {
    console.error('Require error:', e.message);
  }

  return { statusCode: 200 };
};

Fix 6: Switch to Container Images for Complex Dependencies

For very large dependencies (machine learning libraries, headless browsers) that exceed the 250MB layer limit, use container images:

# Dockerfile for Lambda container image
FROM public.ecr.aws/lambda/python:3.11

# Install large dependencies directly in the image (no size limit)
RUN pip install \
    pandas \
    numpy \
    scikit-learn \
    torch \
    --no-cache-dir

# Copy function code
COPY lambda_function.py ${LAMBDA_TASK_ROOT}/

CMD ["lambda_function.handler"]

# Build and push to ECR
aws ecr create-repository --repository-name my-lambda
docker build -t my-lambda .
docker tag my-lambda:latest 123456789.dkr.ecr.us-east-1.amazonaws.com/my-lambda:latest
docker push 123456789.dkr.ecr.us-east-1.amazonaws.com/my-lambda:latest

# Deploy Lambda from container image
aws lambda create-function \
  --function-name my-function \
  --package-type Image \
  --code ImageUri=123456789.dkr.ecr.us-east-1.amazonaws.com/my-lambda:latest \
  --role arn:aws:iam::123456789:role/lambda-role

Lambda container image limits: 10GB (vs 250MB unzipped for layers). Container images also support local testing with docker run.

Still Not Working?

Layer size limits — each layer can be up to 50MB zipped, 250MB unzipped. The total unzipped size of all layers plus your function code must stay under 250MB. For larger dependencies, use container images.

/opt/python vs /opt/python/lib/python3.11/site-packages — both paths are searched, but if your packages have conflicting __init__.py files, the path order matters. Lambda searches /opt/python before /opt/python/lib/pythonX.Y/site-packages.

Layer permissions — layers can be private (your account only), shared with specific accounts, or public. If using a layer from another account, the layer owner must grant permission via aws lambda add-layer-version-permission.

Updating a layer doesn’t update functions automatically — layers are versioned. When you publish a new layer version, you must update each Lambda function to use the new ARN (new version number). Functions keep using the old layer version until explicitly updated.

Hit the 5-layer hard limit — Lambda allows at most five layers per function. If you’ve split dependencies across many small layers for reuse, you can run out of slots once you add observability and secrets extensions. Consolidate domain dependencies into one layer per runtime, and reserve the remaining slots for AWS-published extensions.

Amazon Linux 2 vs Amazon Linux 2023 base image mismatch — older runtimes (Python 3.9, Node.js 16) ship on AL2 with glibc 2.26, while newer runtimes (Python 3.12, Node.js 20) ship on AL2023 with glibc 2.34. A layer built inside public.ecr.aws/lambda/python:3.9 may fail on a python3.12 function with GLIBC_2.34 not found. Always build the layer in the matching base image, or use pip install --platform manylinux_2_28_x86_64 for AL2023 targets.

Layer is attached but pip still resolves the local version — if your deployment package (the function zip) also contains a copy of the same library, that copy takes precedence over /opt/python. Strip your dependencies out of the function zip when you move them to a layer; otherwise version conflicts between the two copies cause ImportError on attribute lookups.

For related AWS Lambda issues, see Fix: AWS Lambda Import Module Error, Fix: AWS Lambda Timeout, Fix: AWS Lambda Cold Start Timeout, and Fix: AWS Lambda Environment Variable Not Set.