Fix: AWS SQS Not Working — Messages Not Received, Duplicate Processing, or DLQ Filling Up

The Problem

Messages are sent to SQS but never received by the consumer:

// Producer sends successfully
await sqs.sendMessage({
  QueueUrl: 'https://sqs.us-east-1.amazonaws.com/123/my-queue',
  MessageBody: JSON.stringify({ orderId: '123' }),
}).promise();
// MessageId returned — but consumer never sees the message

Or the same message is processed multiple times despite being deleted:

// Consumer processes and deletes the message
await sqs.deleteMessage({
  QueueUrl: queueUrl,
  ReceiptHandle: message.ReceiptHandle,
}).promise();
// But the same message appears again 30 seconds later

Or messages go to the Dead Letter Queue immediately without being processed:

DLQ is filling up, but the main queue consumer shows no errors

Or a Lambda function triggered by SQS processes messages out of order.

Why This Happens

SQS has several behaviors that differ from traditional message queues:

• Visibility timeout too short — when a consumer receives a message, it becomes invisible to other consumers for the VisibilityTimeout period. If processing takes longer, the message becomes visible again and is delivered a second time.
• Consumer not polling — ReceiveMessage only returns messages that are currently visible. If polling stops (consumer crash, rate limiting), messages queue up but aren’t processed.
• At-least-once delivery — standard SQS queues guarantee at-least-once delivery. Even with successful deletion, rare cases can deliver the message twice. Your processing must be idempotent.
• DLQ maxReceiveCount too low — if maxReceiveCount is 1, any processing failure sends the message to the DLQ immediately without retry.
• Wrong queue URL or IAM permissions — sending to the wrong URL or missing sqs:ReceiveMessage permissions causes silent failures.

Fix 1: Configure Visibility Timeout Correctly

The visibility timeout must be longer than your maximum processing time:

const { SQSClient, ReceiveMessageCommand, ChangeMessageVisibilityCommand, DeleteMessageCommand } = require('@aws-sdk/client-sqs');

const sqs = new SQSClient({ region: 'us-east-1' });
const QUEUE_URL = 'https://sqs.us-east-1.amazonaws.com/123456789/my-queue';

async function processMessages() {
  const response = await sqs.send(new ReceiveMessageCommand({
    QueueUrl: QUEUE_URL,
    MaxNumberOfMessages: 10,      // Receive up to 10 messages at once
    WaitTimeSeconds: 20,          // Long polling — wait up to 20s for messages
    VisibilityTimeout: 300,       // 5 minutes — must be > max processing time
    AttributeNames: ['All'],
    MessageAttributeNames: ['All'],
  }));

  if (!response.Messages || response.Messages.length === 0) return;

  for (const message of response.Messages) {
    try {
      // If processing might take a long time, extend visibility timeout periodically
      const heartbeat = setInterval(async () => {
        await sqs.send(new ChangeMessageVisibilityCommand({
          QueueUrl: QUEUE_URL,
          ReceiptHandle: message.ReceiptHandle,
          VisibilityTimeout: 300,  // Reset the clock
        }));
      }, 240_000);  // Extend every 4 minutes

      await processMessage(JSON.parse(message.Body));

      clearInterval(heartbeat);

      // Delete only after successful processing
      await sqs.send(new DeleteMessageCommand({
        QueueUrl: QUEUE_URL,
        ReceiptHandle: message.ReceiptHandle,
      }));
    } catch (error) {
      console.error('Processing failed:', error);
      // Don't delete — let SQS retry (message becomes visible after timeout)
    }
  }
}

Set queue visibility timeout in CDK/Terraform:

// AWS CDK
const queue = new sqs.Queue(this, 'MyQueue', {
  visibilityTimeout: Duration.minutes(5),
  receiveMessageWaitTime: Duration.seconds(20),  // Long polling
  retentionPeriod: Duration.days(4),
});

# Terraform
resource "aws_sqs_queue" "my_queue" {
  name                       = "my-queue"
  visibility_timeout_seconds = 300   # 5 minutes
  receive_wait_time_seconds  = 20    # Long polling
  message_retention_seconds  = 345600  # 4 days
}

Fix 2: Implement Long Polling

Short polling returns immediately even with no messages, wasting requests. Use long polling:

// SHORT POLLING (default, wasteful)
// WaitTimeSeconds = 0 — returns immediately if no messages
const response = await sqs.send(new ReceiveMessageCommand({
  QueueUrl: QUEUE_URL,
  WaitTimeSeconds: 0,  // Returns instantly
}));

// LONG POLLING (recommended)
// WaitTimeSeconds = 1-20 — waits up to N seconds for messages
const response = await sqs.send(new ReceiveMessageCommand({
  QueueUrl: QUEUE_URL,
  WaitTimeSeconds: 20,  // Wait up to 20 seconds
  MaxNumberOfMessages: 10,
}));

Continuous polling loop:

async function startConsumer() {
  console.log('Consumer started');

  while (true) {
    try {
      const response = await sqs.send(new ReceiveMessageCommand({
        QueueUrl: QUEUE_URL,
        WaitTimeSeconds: 20,
        MaxNumberOfMessages: 10,
      }));

      if (response.Messages && response.Messages.length > 0) {
        await Promise.all(response.Messages.map(processAndDelete));
      }
    } catch (error) {
      console.error('Poll error:', error);
      await new Promise(r => setTimeout(r, 5000));  // Back off on error
    }
  }
}

async function processAndDelete(message) {
  try {
    await processMessage(JSON.parse(message.Body));
    await sqs.send(new DeleteMessageCommand({
      QueueUrl: QUEUE_URL,
      ReceiptHandle: message.ReceiptHandle,
    }));
  } catch (error) {
    console.error(`Failed to process message ${message.MessageId}:`, error);
    // Message becomes visible again after VisibilityTimeout
  }
}

Fix 3: Configure Dead Letter Queue Properly

A DLQ captures messages that fail processing repeatedly:

// AWS CDK — proper DLQ setup
const dlq = new sqs.Queue(this, 'MyDLQ', {
  queueName: 'my-queue-dlq',
  retentionPeriod: Duration.days(14),  // Keep failed messages 14 days for analysis
});

const mainQueue = new sqs.Queue(this, 'MyQueue', {
  queueName: 'my-queue',
  visibilityTimeout: Duration.minutes(5),
  deadLetterQueue: {
    queue: dlq,
    maxReceiveCount: 3,  // After 3 failed attempts, move to DLQ
  },
});

# Terraform — DLQ configuration
resource "aws_sqs_queue" "dlq" {
  name                      = "my-queue-dlq"
  message_retention_seconds = 1209600  # 14 days
}

resource "aws_sqs_queue" "main" {
  name                       = "my-queue"
  visibility_timeout_seconds = 300

  redrive_policy = jsonencode({
    deadLetterTargetArn = aws_sqs_queue.dlq.arn
    maxReceiveCount     = 3  # Move to DLQ after 3 failures
  })
}

Monitor and replay DLQ messages:

# Check DLQ depth
aws sqs get-queue-attributes \
  --queue-url https://sqs.us-east-1.amazonaws.com/123/my-queue-dlq \
  --attribute-names ApproximateNumberOfMessages

# Redrive DLQ messages back to the main queue (AWS Console or API)
aws sqs start-message-move-task \
  --source-arn arn:aws:sqs:us-east-1:123:my-queue-dlq \
  --destination-arn arn:aws:sqs:us-east-1:123:my-queue \
  --max-number-of-messages-per-second 10

Fix 4: Make Processing Idempotent

Standard SQS delivers at-least-once — the same message may be delivered multiple times. Design processing to handle duplicates:

const redis = require('redis');
const client = redis.createClient();

async function processMessage(message) {
  const messageId = message.MessageId;

  // Check if already processed (using Redis as idempotency store)
  const alreadyProcessed = await client.set(
    `processed:${messageId}`,
    '1',
    { NX: true, EX: 86400 }  // Only set if not exists, expire after 24h
  );

  if (!alreadyProcessed) {
    console.log(`Skipping duplicate message: ${messageId}`);
    return;
  }

  // Process the message
  const body = JSON.parse(message.Body);
  await handleOrder(body.orderId);
}

Database-level idempotency:

-- Postgres: use INSERT ... ON CONFLICT DO NOTHING
INSERT INTO processed_messages (message_id, processed_at)
VALUES ($1, NOW())
ON CONFLICT (message_id) DO NOTHING;

-- Check if it was actually inserted
-- If 0 rows affected, this is a duplicate

Fix 5: Fix Lambda + SQS Integration

When Lambda is triggered by SQS, there are specific behaviors to handle:

// Lambda handler for SQS trigger
import { SQSHandler, SQSRecord } from 'aws-lambda';

export const handler: SQSHandler = async (event) => {
  // event.Records contains all messages in this batch
  const failures: { itemIdentifier: string }[] = [];

  for (const record of event.Records) {
    try {
      await processRecord(record);
    } catch (error) {
      console.error(`Failed to process ${record.messageId}:`, error);

      // Report partial batch failure — only failed messages go back to queue
      failures.push({ itemIdentifier: record.messageId });
    }
  }

  // Return failed message IDs for partial batch failure reporting
  if (failures.length > 0) {
    return { batchItemFailures: failures };
  }
};

async function processRecord(record: SQSRecord) {
  const body = JSON.parse(record.body);
  // Process...
}

Configure the Lambda event source mapping:

// CDK — Lambda + SQS event source
const processFunction = new lambda.Function(this, 'Processor', {
  runtime: lambda.Runtime.NODEJS_20_X,
  handler: 'index.handler',
  code: lambda.Code.fromAsset('lambda'),
  timeout: Duration.minutes(5),  // Must be less than queue's VisibilityTimeout
});

processFunction.addEventSource(new lambdaEventSources.SqsEventSource(mainQueue, {
  batchSize: 10,
  maxBatchingWindow: Duration.seconds(30),  // Wait up to 30s to fill a batch
  reportBatchItemFailures: true,  // Enable partial batch failure handling
}));

Warning: Lambda’s timeout must be less than the SQS visibility timeout. If Lambda times out, the message becomes visible again before Lambda can report the failure, causing duplicate processing.

Fix 6: Use FIFO Queues for Ordered Processing

Standard queues don’t guarantee order. Use FIFO queues when order matters:

// FIFO queue — name must end in .fifo
const fifoQueue = new sqs.Queue(this, 'OrdersQueue', {
  queueName: 'orders.fifo',
  fifo: true,
  contentBasedDeduplication: true,  // Auto-dedup based on message body hash
});

// Sending to FIFO queue — requires MessageGroupId
await sqs.send(new SendMessageCommand({
  QueueUrl: 'https://sqs.us-east-1.amazonaws.com/123/orders.fifo',
  MessageBody: JSON.stringify({ orderId: '123', status: 'shipped' }),
  MessageGroupId: 'order-123',         // All messages for same order in order
  MessageDeduplicationId: 'order-123-shipped-v1',  // Prevent duplicates
}));

FIFO limitations:

• Max 300 transactions/second (3,000 with batching)
• Not available in all AWS regions
• Cannot trigger Lambda with reportBatchItemFailures in some configurations

Still Not Working?

IAM permissions — the consumer role needs sqs:ReceiveMessage, sqs:DeleteMessage, and sqs:ChangeMessageVisibility. The producer needs sqs:SendMessage. Missing permissions cause silent failures (403 errors that look like empty queues):

{
  "Effect": "Allow",
  "Action": [
    "sqs:SendMessage",
    "sqs:ReceiveMessage",
    "sqs:DeleteMessage",
    "sqs:ChangeMessageVisibility",
    "sqs:GetQueueAttributes"
  ],
  "Resource": "arn:aws:sqs:us-east-1:123456789:my-queue"
}

Cross-account or cross-region queues — SQS queue URLs are region-specific. If your producer is in us-east-1 but the queue is in eu-west-1, use the correct region in the SQSClient configuration, and ensure the queue policy allows cross-account access.

ApproximateNumberOfMessages shows 0 but messages aren’t processing — messages may be in flight (currently invisible, being processed). Check ApproximateNumberOfMessagesNotVisible. If it’s high, your consumers are receiving messages but not deleting them (processing is stuck or failing silently).

For related AWS issues, see Fix: AWS Lambda Timeout and Fix: Celery Task Not Received.