Fix: Kubernetes Pod stuck in Pending state

The Error

You deploy a Pod and it stays in Pending status indefinitely:

$ kubectl get pods
NAME                     READY   STATUS    RESTARTS   AGE
my-app-7b4f5c8d9-xk2lm  0/1     Pending   0          10m

Running kubectl describe pod shows events like:

Events:
  Warning  FailedScheduling  default-scheduler  0/3 nodes are available:
  1 node(s) had untolerated taint {node.kubernetes.io/not-ready: },
  2 node(s) didn't match Pod's node affinity/selector,
  3 Insufficient cpu.

Or:

Warning  FailedScheduling  0/5 nodes are available: 5 Insufficient memory.

Warning  FailedScheduling  pod has unbound immediate PersistentVolumeClaims.

The Kubernetes scheduler cannot find a node to place the Pod on. The Pod sits in Pending until the scheduling constraint is resolved.

Why This Happens

A Pod enters Pending when the scheduler cannot find a suitable node. The scheduler runs a two-phase loop: a filtering phase that eliminates nodes that cannot run the Pod, and a scoring phase that ranks the survivors. If filtering removes every node, the Pod never reaches scoring and stays Pending until the constraint changes.

Filtering checks the following predicates, in this rough order:

1. Resource requests. Does any node have enough CPU and memory in its allocatable budget?
2. Node selectors and affinity. Does the Pod require specific node labels that match a real node?
3. Taints and tolerations. Is the node tainted, and does the Pod tolerate every taint with a matching effect?
4. PersistentVolumeClaims. Are the requested volumes bound, and is the volume’s zone reachable from a candidate node?
5. Resource quotas. Has the namespace exceeded its CPU, memory, or Pod count quota?
6. Pod topology constraints. Are there spread constraints that cannot be satisfied with the current node pool?

The kubectl describe pod events tell you exactly which predicate failed and on how many nodes. The format 0/N nodes are available: X reasons lists every distinct rejection reason and the count of nodes that hit it. Always read this block first — the rest of this article is unblocking those specific reasons.

One more subtlety. The scheduler does not consider current node utilization, only the sum of resource requests from already-scheduled Pods. A node sitting at 5% CPU usage but with requests totaling 95% will still reject your Pod. This is why oversized requests are the single most common cause of phantom Pending Pods.

In Production: Incident Lens

In production, Pending Pods are a capacity event. Existing Pods continue to serve traffic, so user-facing SLOs usually hold for the first few minutes, but anything that needs a new Pod stalls — rolling deploys, HPA scale-out, node replacements, and Job runs all freeze. The blast radius is “no new Pods of any kind can start” until the constraint clears.

The standard monitoring signal is kube_pod_status_phase{phase="Pending"} exceeding a threshold for more than five minutes, ideally split by namespace so a stuck Job in one team’s namespace does not page the whole org. Pair it with cluster_autoscaler_unschedulable_pods_count — if both are non-zero and growing, the autoscaler tried to add a node and failed (quota, AMI, instance type unavailable in the zone). The recovery sequence is: confirm describe pod shows a real resource shortage rather than a transient unschedulable flag, manually scale the node group if the autoscaler is wedged, and cordon any node that is reporting pressure to stop the scheduler from retrying against it.

Postmortem preventives are almost always the same: enforce resource request budgets per namespace via LimitRange, run a descheduler to rebalance long-running Pods that drifted into hot nodes, and set maxUnavailable and maxSurge on rolling deploys so the deploy does not deadlock when capacity is tight. Tracking median request vs actual usage per workload is the highest-signal metric for catching oversized requests before they fill the cluster.

Fix 1: Fix Insufficient CPU or Memory

The most common cause. No node has enough free resources:

# Check the events
kubectl describe pod my-app-7b4f5c8d9-xk2lm | grep -A 10 Events

# Check node resource usage
kubectl top nodes
kubectl describe nodes | grep -A 5 "Allocated resources"

Reduce resource requests:

# Before — requesting too much
resources:
  requests:
    cpu: "4"
    memory: "8Gi"

# After — right-sized
resources:
  requests:
    cpu: "500m"       # 0.5 CPU cores
    memory: "512Mi"
  limits:
    cpu: "1"
    memory: "1Gi"

Check what is consuming resources on the nodes:

# List all pods sorted by CPU request
kubectl get pods --all-namespaces -o custom-columns=\
"NAMESPACE:.metadata.namespace,NAME:.metadata.name,CPU:.spec.containers[*].resources.requests.cpu,MEM:.spec.containers[*].resources.requests.memory" \
| sort -k3 -h

Scale the cluster (add more nodes):

# EKS
eksctl scale nodegroup --cluster my-cluster --nodes 5 --name my-nodegroup

# GKE
gcloud container clusters resize my-cluster --num-nodes 5

# AKS
az aks scale --resource-group myRG --name myCluster --node-count 5

Pro Tip: Set resource requests to what the Pod actually needs under normal load, and limits to the maximum it should ever use. Overly generous requests waste cluster capacity. Use kubectl top pods to measure actual usage before setting requests.

Fix 2: Fix Node Selector and Affinity Issues

The Pod requires specific node labels that no node has:

# Pod requires nodes labeled with gpu=true
spec:
  nodeSelector:
    gpu: "true"

Check available node labels:

kubectl get nodes --show-labels
# or
kubectl get nodes -L gpu

Fix: Add the label to a node:

kubectl label nodes my-node gpu=true

Fix: Remove or update the node selector:

spec:
  # Remove nodeSelector if not needed
  # Or use nodeAffinity for soft preferences
  affinity:
    nodeAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 1
          preference:
            matchExpressions:
              - key: gpu
                operator: In
                values: ["true"]

preferredDuringScheduling is a soft constraint — the scheduler prefers matching nodes but will use others if none match. requiredDuringScheduling is hard — the Pod stays Pending until a matching node exists.

Fix 3: Fix Taints and Tolerations

Nodes might have taints that prevent scheduling:

# Check node taints
kubectl describe node my-node | grep Taints
# Taints: node.kubernetes.io/not-ready:NoSchedule

Common taints:

Add tolerations to the Pod:

spec:
  tolerations:
    - key: "gpu"
      operator: "Equal"
      value: "true"
      effect: "NoSchedule"

Remove a taint from a node:

kubectl taint nodes my-node gpu=true:NoSchedule-
# The trailing - removes the taint

For control plane nodes (single-node clusters):

kubectl taint nodes --all node-role.kubernetes.io/control-plane-

Common Mistake: Running a single-node cluster (like Minikube or kind) and wondering why Pods are Pending. Control plane nodes are tainted by default. Either add tolerations or remove the taint.

Fix 4: Fix PersistentVolumeClaim Issues

If the Pod uses a PVC that is not bound:

Warning  FailedScheduling  pod has unbound immediate PersistentVolumeClaims

Check PVC status:

kubectl get pvc
# NAME        STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS
# my-data     Pending                                       standard

Common causes:

# No PersistentVolume matches the PVC
kubectl get pv

# StorageClass doesn't exist
kubectl get storageclass

# The storage provisioner is not running
kubectl get pods -n kube-system | grep provisioner

Fix: Create a PersistentVolume:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: my-pv
spec:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: /data/my-pv

Fix: Use a valid StorageClass:

# List available storage classes
kubectl get storageclass

# Update PVC to use an existing storage class

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-data
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: standard  # Must match an existing StorageClass
  resources:
    requests:
      storage: 10Gi

For cloud providers, check the volume zone:

# The PVC might request a zone where no nodes exist
kubectl describe pvc my-data

Volumes are often zone-specific. If your node is in us-east-1a but the volume is in us-east-1b, the Pod cannot be scheduled.

Fix 5: Fix ResourceQuota Limits

The namespace might have resource quotas that prevent scheduling:

kubectl describe resourcequota -n my-namespace
# Name:       my-quota
# Resource    Used    Hard
# --------    ----    ----
# cpu         3800m   4000m
# memory      7Gi     8Gi
# pods        19      20

Fix: Increase the quota:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: my-quota
  namespace: my-namespace
spec:
  hard:
    cpu: "8"
    memory: "16Gi"
    pods: "40"

Fix: Reduce resource requests on existing Pods:

# Find Pods using the most resources
kubectl top pods -n my-namespace --sort-by=cpu

Fix: Clean up unused Pods:

# Delete completed jobs
kubectl delete jobs --field-selector status.successful=1 -n my-namespace

# Delete evicted pods
kubectl delete pods --field-selector status.phase=Failed -n my-namespace

Fix 6: Fix LimitRange Issues

A LimitRange might set minimum resource requirements higher than your Pod specifies:

kubectl describe limitrange -n my-namespace

# If LimitRange requires minimum 256Mi memory but your Pod requests 128Mi,
# the Pod will not be admitted
Type        Resource  Min     Max    Default Request  Default Limit
----        --------  ---     ---    ---------------  -------------
Container   cpu       100m    2      200m             500m
Container   memory    256Mi   4Gi    256Mi            512Mi

Fix: Increase your Pod’s resource requests to meet the minimum:

resources:
  requests:
    cpu: "100m"
    memory: "256Mi"  # Must meet the LimitRange minimum

Fix 7: Fix Pod Topology Spread Constraints

Spread constraints can prevent scheduling if there are not enough nodes:

spec:
  topologySpreadConstraints:
    - maxSkew: 1
      topologyKey: kubernetes.io/hostname
      whenUnsatisfiable: DoNotSchedule  # This makes it a hard constraint
      labelSelector:
        matchLabels:
          app: my-app

If you have 3 nodes and already have 1 Pod on each, a 4th Pod cannot satisfy maxSkew: 1 with DoNotSchedule.

Fix: Use ScheduleAnyway:

whenUnsatisfiable: ScheduleAnyway  # Soft constraint — best effort

Fix: Add more nodes or reduce replicas.

Fix 8: Debug Scheduling with Events and Logs

Check Pod events:

kubectl describe pod <pod-name> | tail -20

Check scheduler logs:

kubectl logs -n kube-system -l component=kube-scheduler --tail=50

Simulate scheduling:

# Dry-run to see where a Pod would be placed
kubectl get nodes -o json | jq '.items[] | {name: .metadata.name, allocatable: .status.allocatable}'

Force-reschedule all Pending Pods:

# Delete and let the Deployment recreate them
kubectl delete pod <pod-name>

Still Not Working?

Check for node cordons. A node might be cordoned (marked as unschedulable):

kubectl get nodes
# NAME      STATUS                     ROLES    AGE
# node-1    Ready,SchedulingDisabled   <none>   30d

# Uncordon the node
kubectl uncordon node-1

Check for PodDisruptionBudgets (PDBs). A PDB might prevent rescheduling during rolling updates.

Check for init containers. If an init container cannot complete (e.g., waiting for a service), the Pod stays in a Pending-like state (Init:0/1).

Check the cluster autoscaler decision log. If you run an autoscaler, it logs why it did or did not add a node. The most common silent failure is hitting a cloud quota (EC2 vCPU limit, GCE in-use IP addresses, AKS subnet exhaustion). The Pod stays Pending forever because the autoscaler asks for capacity, gets rejected, backs off, and never retries. Check the autoscaler Pod logs or the cloud console quota page:

kubectl logs -n kube-system -l app=cluster-autoscaler --tail=200 | grep -iE "quota|exceed|not avail"

Check for instance type availability in the zone. Spot or specific GPU instance types are not always available in every zone. The autoscaler may try, fail, and silently skip. Either expand the node group to multiple zones or fall back to a more common instance family.

Check for kubelet not registering the node. A node that has booted but whose kubelet cannot reach the API server (bad token, expired certificate, security group blocking 10250) will not appear in kubectl get nodes. Capacity exists on the VM but Kubernetes cannot use it. Restart kubelet and check /var/log/kubelet.log for handshake failures.

Check for stuck Pending evictions on a parent node. When a node enters NotReady, the controller manager tries to evict its Pods after the toleration window. If those evicted Pods have anti-affinity rules that prevent them from landing elsewhere, they stay Pending while the original node sits cordoned indefinitely.

For Pods that start but immediately crash, see Fix: Kubernetes Pod CrashLoopBackOff. For image pull failures, see Fix: Kubernetes ImagePullBackOff. For OOM kills, see Fix: Kubernetes Pod OOMKilled. For namespace quota errors that surface as admission rejections rather than Pending Pods, see Fix: Kubernetes resource quota exceeded.