Taming Celery queues without losing your mind

Every Django project I've shipped eventually grows a queue. Emails, webhooks, report generation — anything that shouldn't block a request ends up in Celery. It works great until the day a downstream API goes down and the retry backlog quietly eats your Redis memory.

Separate queues by failure domain

The first mistake is routing everything through a single celery queue. When one task type starts failing and retrying aggressively, it starves every other task behind it. Splitting queues by how a task is allowed to fail — transactional, best-effort, reporting — means a broken webhook integration can't block password reset emails.

Bound your retries, always

max_retries with exponential backoff is table stakes, but the detail that matters is a hard ceiling on total retry time, not just retry count. A task that retries 10 times with a 5-minute cap behaves very differently from one with a 30-minute cap once the queue is under real load.

Idempotency is not optional

If a task can be retried, it will eventually run twice — a worker will get killed mid-execution, a network blip will cause a duplicate ack. Design every task as if it might run twice: use get_or_create, unique constraints, and idempotency keys rather than trusting retry counts.

Alert on queue depth, not just failures

By the time individual task failures show up in your error tracker, the backlog is often already hours deep. A simple depth-over-time alert on each queue catches the slow creep long before users notice.

None of this is exotic — it's mostly discipline. But discipline is exactly what's missing the first time a queue gets put into production under a deadline.