Fix: Celery Task Not Received or Not Executing

The Error

You send a Celery task, but nothing happens. The task either never appears in the worker output, or you see:

Received unregistered task of type 'myapp.tasks.process_order'

Or the task is sent successfully but the worker shows no activity:

result = process_order.delay(order_id=123)
print(result.id)  # Returns a task ID
# But the worker never picks it up

The worker just sits idle, or the task gets silently dropped.

Why This Happens

Celery has multiple components that must all be configured correctly: the broker (Redis or RabbitMQ) that holds messages, the worker that processes them, and the application that sends them. A failure at any point in this chain causes tasks to be lost or ignored.

Common causes include the worker not connecting to the same broker as the sender, task names not matching between sender and worker, the autodiscovery mechanism not finding your task modules, or the worker listening on a different queue.

The trap with Celery is that the sender almost always succeeds. task.delay() writes a JSON message into the broker and returns an AsyncResult with an ID — that ID exists whether or not any worker will ever pick the message up. So the symptom (no execution) and the cause (no consumer matching the queue, or no registration matching the name) are separated by an entire system boundary. You cannot tell from the sender’s side whether the message was received, only whether it was queued. That is why this bug eats half a day if you debug from the wrong end. Always start the diagnosis at the worker, not the sender.

The other recurring trap is environment drift. The web process and the worker process are two separate Python interpreters with two separate sets of environment variables, two separate imports, and often two separate deploy targets. If CELERY_BROKER_URL is defined in the web’s .env but missing from the worker’s systemd unit, the worker silently falls back to amqp://localhost// and you sit there wondering why messages vanish. Treat the worker as a different machine from your web tier even when they share a host.

Diagnostic Timeline

Run these steps in order. They isolate the failure layer in under 15 minutes.

Minute 0 — Confirm the message was actually queued. Run redis-cli llen celery (or rabbitmqctl list_queues name messages). If the count is zero immediately after .delay(), the sender is misconfigured or pointing at a different broker. If the count is non-zero, the message exists and the worker is the problem.

Minute 2 — Ask the worker what it knows. Run celery -A myproject inspect registered. This prints every task name the worker has imported. If your task name is missing, the worker never autodiscovered the module. If the name is present but differs from what the sender uses (myapp.tasks.foo vs tasks.foo), you have a naming mismatch.

Minute 4 — Ask the worker which queues it consumes. Run celery -A myproject inspect active_queues. Compare against the queue your sender targets. If the sender writes to priority but the worker only consumes celery, the message sits forever.

Minute 7 — Verify both sides see the same broker. In a shell on the sender host, run python -c "from myproject.celery import app; print(app.conf.broker_url)". Repeat on the worker host. If the URLs differ — different host, different database number, different vhost — that is the bug.

Minute 10 — Ping the worker. Run celery -A myproject inspect ping. A response means the worker process is alive and reachable through the broker. No response means the worker is dead, started against a different broker, or behind a firewall.

Minute 12 — Check for the visibility-timeout silent redelivery. If tasks appear to run but never finish, the worker may be exceeding Redis’s default 1-hour visibility timeout. Redis redelivers the message, the worker picks it up again, and you see the same task ID appearing twice in flower. Set broker_transport_options = {'visibility_timeout': 43200} for long tasks.

Minute 15 — Rule out task_always_eager. Print app.conf.task_always_eager from the sender’s shell. If it is True, your tasks run synchronously in the web process and never touch the broker at all. This is the single most embarrassing root cause and the easiest to confirm.

Fix 1: Verify Broker Connection

Both the sender and worker must connect to the same broker. Check your Celery configuration:

# celery.py or config
app = Celery('myproject')
app.config_from_object('django.conf:settings', namespace='CELERY')

# settings.py
CELERY_BROKER_URL = 'redis://localhost:6379/0'

Test the broker connection:

from celery import Celery
app = Celery('test', broker='redis://localhost:6379/0')
print(app.connection().ensure_connection(max_retries=3))

If using Redis, verify it’s running:

redis-cli ping  # Should return PONG

Common mistakes:

• Using redis://localhost in the app but the worker is on a different machine
• Using different Redis databases (/0 vs /1)
• The broker URL is set in environment variables that aren’t loaded in the worker’s environment

Fix 2: Fix Task Autodiscovery

Celery’s autodiscover_tasks() finds tasks automatically, but it needs to know where to look:

# celery.py
app = Celery('myproject')
app.config_from_object('django.conf:settings', namespace='CELERY')
app.autodiscover_tasks()  # Searches INSTALLED_APPS for tasks.py

For Django, this searches each app in INSTALLED_APPS for a tasks.py file. If your tasks are in a different file, they won’t be discovered:

# This is found automatically
myapp/tasks.py

# This is NOT found automatically
myapp/celery_tasks.py
myapp/workers/tasks.py

To include non-standard task locations:

app.autodiscover_tasks(['myapp.workers', 'myapp.background'])

Or register tasks explicitly:

# Include specific modules
app.conf.include = ['myapp.workers.tasks', 'myapp.background.jobs']

Pro Tip: Run celery -A myproject inspect registered to see all tasks the worker knows about. If your task isn’t in the list, it hasn’t been discovered. Compare this list with what your sender expects.

Fix 3: Fix Task Name Mismatches

Celery generates task names based on the module path. If the sender and worker import tasks differently, the names won’t match:

# If the task is defined in myapp/tasks.py
@app.task
def process_order(order_id):
    pass

# Celery names it: 'myapp.tasks.process_order'

Problems arise when:

# Sender uses absolute import
from myapp.tasks import process_order
# Task name: 'myapp.tasks.process_order'

# But worker's autodiscovery finds it as
# Task name: 'tasks.process_order'

Set an explicit name to avoid ambiguity:

@app.task(name='process_order')
def process_order(order_id):
    pass

Or use send_task with the exact name:

app.send_task('myapp.tasks.process_order', args=[123])

Check registered task names on the worker:

celery -A myproject inspect registered

Fix 4: Verify the Worker Is Running

This sounds obvious, but it’s a common issue. The worker must be running and connected:

# Start the worker
celery -A myproject worker --loglevel=info

# Check if it's running
celery -A myproject inspect ping

The worker output should show:

[config]
.> app:         myproject:0x...
.> transport:   redis://localhost:6379/0
.> results:     redis://localhost:6379/0
.> concurrency: 4 (prefork)

[queues]
.> celery    exchange=celery(direct) key=celery

If you see no output or connection errors, the worker can’t reach the broker. Check the broker URL and network connectivity.

For production, run the worker as a systemd service:

# /etc/systemd/system/celery.service
[Unit]
Description=Celery Worker
After=network.target

[Service]
Type=forking
User=celery
WorkingDirectory=/opt/myproject
ExecStart=/opt/myproject/venv/bin/celery -A myproject multi start worker1 \
  --loglevel=info --logfile=/var/log/celery/%n%I.log
ExecStop=/opt/myproject/venv/bin/celery multi stopwait worker1

Fix 5: Fix Queue Routing

If the task is routed to a specific queue but the worker listens on the default queue, the task is never picked up:

# Task routed to 'priority' queue
@app.task(queue='priority')
def urgent_task():
    pass

# Or via routing configuration
app.conf.task_routes = {
    'myapp.tasks.urgent_task': {'queue': 'priority'},
    'myapp.tasks.*': {'queue': 'default'},
}

The worker must explicitly consume from that queue:

# Only listens to 'celery' (default) queue
celery -A myproject worker

# Listen to specific queues
celery -A myproject worker -Q celery,priority

# Listen to all queues
celery -A myproject worker -Q celery,priority,default

Check which queues have pending messages:

# Redis
redis-cli llen celery    # Default queue
redis-cli llen priority  # Custom queue

# RabbitMQ
rabbitmqctl list_queues name messages

Common Mistake: Sending a task to a queue that no worker consumes from. The messages pile up in the broker but are never processed. Always verify that at least one worker is listening on every queue you route tasks to.

Fix 6: Fix Serialization Issues

Celery serializes task arguments before sending them to the broker. If the arguments can’t be serialized, the task fails silently or raises an error:

# This fails with JSON serializer (default in Celery 4+)
@app.task
def process(data):
    pass

process.delay(data=datetime.now())  # datetime isn't JSON serializable

Fix by converting arguments to serializable types:

process.delay(data=datetime.now().isoformat())

Or change the serializer:

app.conf.task_serializer = 'pickle'  # Handles more types
app.conf.accept_content = ['pickle', 'json']

Warning: Using pickle is a security risk if your broker is accessible to untrusted clients. Stick with JSON and convert your data to serializable formats.

Check serialization compatibility:

import json
try:
    json.dumps(your_task_arguments)
except TypeError as e:
    print(f"Not JSON serializable: {e}")

Fix 7: Disable task_always_eager for Production

task_always_eager runs tasks synchronously in the current process, bypassing the broker entirely. It’s useful for testing but causes confusion in production:

# settings.py
CELERY_TASK_ALWAYS_EAGER = True  # Tasks run immediately, no worker needed

If this is enabled in production, tasks run in the web process and never reach the worker. Disable it:

# settings.py
CELERY_TASK_ALWAYS_EAGER = False  # Default
CELERY_TASK_EAGER_PROPAGATES = False

For testing, use task_always_eager only in test settings:

# test_settings.py
CELERY_TASK_ALWAYS_EAGER = True
CELERY_TASK_EAGER_PROPAGATES = True  # Propagate exceptions in tests

Fix 8: Tune Concurrency and Prefetch Settings

Workers may appear to not process tasks if they’re stuck on long-running tasks and the prefetch limit prevents new tasks from being fetched:

# Worker takes all available tasks but processes slowly
app.conf.worker_prefetch_multiplier = 4  # Default: fetches 4 × concurrency tasks

For long-running tasks, reduce prefetch:

app.conf.worker_prefetch_multiplier = 1  # Fetch one task at a time per worker process

Or for tasks with unpredictable duration:

celery -A myproject worker --concurrency=4 -Ofair

The -Ofair flag distributes tasks to workers that are actually free, rather than prefetching to all workers equally.

Check worker status:

celery -A myproject inspect active     # Currently executing
celery -A myproject inspect reserved   # Prefetched, waiting
celery -A myproject inspect stats      # General statistics

Still Not Working?

• Check the result backend. If you’re using result.get() and the result backend isn’t configured, the call hangs forever. Set CELERY_RESULT_BACKEND or use result.get(timeout=10).

• Look for import errors. If a task module has an import error, the worker skips it silently. Check worker startup output for import warnings.

• Verify environment variables. If your tasks depend on env vars, ensure they’re available in the worker’s environment, not just the web process.

• Check for task rate limits. @app.task(rate_limit='10/m') limits execution to 10 per minute. Tasks beyond the limit are queued but delayed.

• Monitor with Flower. Install flower (pip install flower) and run celery -A myproject flower for a web dashboard showing task status, worker health, and queue depths.

• Check broker visibility timeout. For Redis, if a task takes longer than the visibility timeout (1 hour default), Redis redelivers it. Set broker_transport_options = {'visibility_timeout': 43200} for long tasks.

• Restart workers after every deploy. Celery workers cache imported modules at startup. If you deploy new task code without restarting the worker, the worker keeps running the old code and may even reject new task names as “unregistered.” Wire celery multi restart or systemctl restart celery into your deploy script.

• Look for forked-child poisoning on macOS and Python 3.8+. The default prefork pool uses fork(), which is unsafe with some C extensions (notably gRPC and some database drivers). The child dies silently on first use and you see no tasks executing. Switch to the thread pool with --pool=threads or the gevent pool to confirm.

• Verify RabbitMQ vhost permissions. A user with read but not write permission on the vhost can publish but not ack, so messages reappear in the queue after every fetch. Run rabbitmqctl list_user_permissions <user> and verify configure/write/read are all set to .*.