Celery and RabbitMQ: Getting Long Running Work Out of the Request Cycle

The incident that taught me this lesson took four minutes. A client emailed a 2GB video to a feature built for short clips, the upload handler tried to process it inline, the worker pool saturated, and every request behind it queued until the load balancer started returning errors. The app was "down" and not one line of code had crashed.

A request handler should do three things: validate input, write a few rows, return fast. The moment it also resizes images, calls a third party API, sends emails or runs a model, you have built the same time bomb we had. I have since moved that class of work into Celery and RabbitMQ on several production systems: image pipelines, notification fan outs, data syncs, AI workloads. Here is the field guide I wish I had before that afternoon.

The mental model

• Producer (your web app) drops a small message: "process upload 123".
• Broker (RabbitMQ) holds it durably until a worker is ready.
• Worker (Celery) does the slow thing with retries and logging.
• The user gets an instant response and sees status update when the work lands.

The web tier goes back to doing what it is good at: being fast. The worker tier absorbs the chaos, and chaos is what it is built for.

The five rules that keep you out of trouble

1. Pass IDs, not objects

Serialize a primary key, not a model instance or a dict snapshot. The task should re-fetch fresh state when it runs, because the world may have changed between enqueue and execute. A user can cancel an order in the seconds before the confirmation task fires. Fresh reads make that a non event instead of a support ticket. This also keeps messages tiny, which keeps the broker healthy under bursts.

2. Every task must be idempotent

Retries mean your task will run twice eventually. Not might. Will. Design for it: upsert instead of insert, check then act inside a transaction, or key side effects on a deterministic identifier, like "email for order 42 shipped, already sent? skip". Idempotency turns retries from a hazard into a free reliability feature, and it is the single property I check first when reviewing anyone's task code.

3. Retry with backoff, but classify failures first

Network timeout? Retry with exponential backoff and jitter. Validation error? Do not retry. It will fail identically forever and clog your queue while doing it. Split exceptions into retryable and terminal, and send terminal failures somewhere a human actually looks: Sentry, a dead letter queue, an admin table with a red badge. A queue silently recycling poison messages is one of the most expensive quiet failures in backend engineering.

4. Separate queues by latency class

One queue is a trap. A burst of heavy video jobs will starve the password reset email a user is staring at their inbox waiting for. We run three queues minimum: interactive for anything a human is actively awaiting, default for routine work, heavy for media, ML and exports, each with its own worker pool. Capacity planning becomes per class instead of global, and one noisy feature can no longer take hostages.

5. Watch queue depth, not just worker health

Workers being "up" means nothing if the queue grows faster than it drains. The two numbers that predict incidents before users feel them: queue depth and oldest message age. Alert on both. In practice, an age alert at two minutes on the interactive queue has paged me exactly when it should and never when it should not.

The pattern for user facing jobs

For anything the user watches, an upload processing or a report generating:

1. Create a row with status pending and return its ID immediately.
2. The worker walks it through processing to done or failed, writing progress along the way.
3. The frontend polls or subscribes over a websocket.

Users tolerate waiting. They do not tolerate not knowing. A visible progress state beats a fast but fragile synchronous call every single time, and it turns timeout bugs into UX states you designed on purpose.

When you do not need all this

A cron style cleanup or one low volume email? FastAPI's BackgroundTasks or a simple database backed job table is enough, and adding a broker would just be operational weight. Reach for Celery and RabbitMQ when you need retries, fan out, rate control or independent scaling. Reach for the simple thing when you do not. Knowing which situation you are in is most of the skill.

The takeaway

Slow work in the request cycle is a countdown you have not looked at yet. Queues are not architecture astronautics; they are the difference between a bad upload being a log line and being an outage.

Drowning requests in slow work? I can help you untangle it. Contact me.