One Loop, Many Slots: Cutting Polling Load by 95%

Superclerk runs agents that can reply to conversations, react to app events, sync external data, and run scheduled automations. Some of that work should happen after the original request is gone. We do not want a user request, webhook, or schedule tick to sit open while the agent calls providers, retries, and writes results.

So the system has two separate processes:

Process	What it does
Web/API process	Receives product events and saves work into the queue. Then it can return quickly.
Worker process	Runs in the background, reads queued work, performs it, and records the result.

The queue is not a magic service. In this system, the durable queue is backed by Postgres. Postgres stores what work exists, what is waiting, what is running, what finished, and what should retry later.

The pieces are simple:

Piece	Meaning
Job	A durable record that says: run this kind of work later with this input. It is data, not a process.
Handler	A normal async function that knows how to do one kind of job.
Worker	A separate backend service process that keeps running, claims jobs, and calls handlers. It is not a browser Web Worker. It is not the job.
Slot	One execution seat inside the worker. 25 slots means the worker can run up to 25 jobs at the same time for that queue.
Loop	The small repeat cycle inside the worker: check open slots, claim jobs, start handlers, wait.
Postgres	The durable store for waiting jobs, running jobs, finished jobs, retries, and waits.

From Product Event To Finished Work

Start at the beginning: something happens in the product. Maybe a Telegram message arrives, a schedule fires, or an integration event arrives.

The Web/API process decides what should happen next. If the agent needs to send a reply later, the Web/API process does not send it directly. It writes a job to the queue:

await queue.add({
  type: "sendReply",
  input: { conversationID: "conv_123", message: "Done." },
  state: "waiting",
})

That creates a durable record in Postgres. A simplified version looks like this:

{
  id: "job_123",
  type: "sendReply",
  input: { conversationID: "conv_123", message: "Done." },
  state: "waiting",
  attempts: 0,
}

Nothing has been sent yet. This record only means: “there is work waiting.”

Separately, the Worker process is already running. Its loop keeps asking one question:

how many open slots do I have?

If the worker has open slots, it asks Postgres for that many waiting jobs. Claiming a job means Postgres marks it as owned by this worker attempt, so another worker does not run the same attempt at the same time:

const jobs = await postgres.claimJobs({ limit: openSlots })

Now the worker has real jobs to run. For each claimed job, it calls the normal application function for that job type:

async function sendReply(input) {
  await telegram.sendMessage(input.conversationID, input.message)
}

for (const job of jobs) {
  await handlers[job.type](job.input)
}

That handler does the real product work: call a provider, send a message, sync data, write results, or whatever that job type means.

When the handler succeeds, the worker marks the job completed in Postgres. If it throws, the worker records the error and either schedules a retry or marks the job failed after attempts are exhausted.

The full flow is:

product event happens
Web/API process writes a job
Postgres stores the job as waiting
Worker process claims the job
Worker calls the handler function
Handler does the product work
Worker records completed, retry, or failed

This was not a classic “work is stuck” failure. Jobs were completing. Postgres was not deadlocked. The problem was subtler: the worker had turned concurrency into database polling load.

The mistake was that every slot was also asking Postgres for work.

The Bug In Plain English

Before the fix, each slot behaved like a small independent worker:

Before

25 slots
  -> each slot asks Postgres: "is there work for me?"
  -> the same question gets repeated over and over
  -> Postgres spends too much time coordinating workers

After the fix, one loop watches all 25 slots for that queue:

After

25 slots
  -> one loop counts how many slots are free
  -> if 3 slots are free, it asks Postgres for 3 jobs
  -> if 0 slots are free, it does not ask Postgres for more jobs

The worker can still run up to 25 jobs at once. The fix did not reduce capacity. It removed duplicate asking.

That is the whole lesson: more capacity should mean more jobs can run, not more loops repeatedly asking the database the same question.

What Changed

We changed the worker from “each slot asks Postgres” to “one loop fills open slots.”

The fixed worker keeps a small local list of jobs it is already running:

runningJobs = jobs already in flight
openSlots = slotLimit - runningJobs

Then the loop does the obvious thing. It is not a tight spin; it waits until a running job finishes or the poll timer fires.

If 25 jobs are already running, do not ask Postgres for more.
If 3 slots are open, ask Postgres for 3 jobs.
Mark those jobs as running and call their handlers.
When a handler finishes, completes, fails, or retries, that slot opens again.

The fix was not “poll less” as a magic setting. It was “ask only when there is room to run what we pick up.”

The Production Signal

The useful signal came from three places: CloudWatch logs, PlanetScale Query Insights, and ECS CPU.

The database was Postgres, accessed through the normal application connection pool. We used PlanetScale Query Insights to measure what the worker was making Postgres do, not just what the worker logs said.

CloudWatch showed repeated worker-loop warnings about work ownership. That warning was a symptom, not the root cause. The stronger proof was database shape: PlanetScale showed the worker repeatedly asking the database coordination questions even when useful work was bounded by the number of available execution slots.

Here is how to read the main metrics:

Metric	What it means
Coordination reads	How often workers touched the database just to coordinate work.
Work pickup checks	How often workers asked whether runnable work existed.
Active-work checks	How often workers rechecked current execution state.
Stale-ownership checks	The cleanup tax for work that might have lost ownership.
cleanup DB time	Database time spent on maintenance scans.
Worker-loop warnings/errors	The worker telling us its own control loop was noisy.

The 6-Hour Fixed Window

The cleanest comparison is the fixed 6-hour window before and after the rollout. It is short enough to avoid unrelated daily traffic changes and long enough to show the mechanism clearly.

Metric	Pre-fix 6h	Post-fix 6h	Reduction
Coordination reads	20.6M	564k	36.5x less, 97.3% lower
Work pickup checks	6.75M	210k	32.1x less, 96.9% lower
Active-work checks	7.07M	349k	20.3x less, 95.1% lower
Stale-ownership checks	7.07M	349k	20.3x less, 95.1% lower
cleanup scan DB time	1,063s	52s	20.4x less, 95.1% lower
cleanup data scanned	1.29B	93.3M	13.8x less, 92.8% lower
worker-loop warnings/errors	33	0	fixed
job final states	1,521 completed, 0 failed	1,671 completed, 0 failed	no throughput loss

The important line is not just that one number improved. Every hot path tied to the old polling shape moved in the same direction.

The 24-Hour Confirmation

The 24-hour view confirmed the same pattern at a larger window.

Metric	Pre-fix 24h	Post-fix 24h	Change
Coordination reads	80.7M	2.23M	36.2x less
Work pickup checks	26.4M	832k	31.8x less
Active-work checks	27.7M	1.38M	20.1x less
Stale-ownership checks	27.7M	1.38M	20.1x less
cleanup scan DB time	4,031s	220s	18.3x less
cleanup data scanned	5.02B	395M	12.7x less
worker-loop warnings/errors	179	0	fixed

ECS worker CPU moved with the database load:

00:00 PDT before fix: 98.2%
01:00 PDT before fix: 98.1%
02:00 PDT rollout hour: 59.1%
03:00-08:00 PDT after fix: roughly 33-37%

The worker did not merely complete a similar number of jobs. It stopped asking the database unnecessary questions.

Why This Did Not Lose Throughput

The obvious concern is that fewer polls might mean slower work pickup.

That would be true if polling were the only way to notice free capacity. The fixed worker does not rely on that. It wakes itself when a run settles.

If one run completes, the loop immediately recomputes free slots and tries to refill them. The poll timer is still there, but it is a safety net. It is not the main engine.

That is why the fix reduced database work without starving real work.

How The Worker Refills Slots

A worker should not wait for a long poll interval after a slot opens. If one job completes, the loop should notice the open slot and fill it.

The fixed loop waits on two things:

an active run settling
the poll timer firing

If a run settles first, the loop refills capacity. If no work finishes, the timer checks again later.

This also matters for partitioned work. A long-running partition should not prevent unrelated partitions from using free capacity. The regression tests name those behaviors directly:

worker loop refills free slots while another partition is still running
worker loop refills after concurrency frees without waiting for the poll interval

Making Capacity A Real Invariant

The polling fix exposed a second boundary: work pickup must respect capacity as one coherent decision.

If two pickup attempts from the same worker check capacity at the same time, they can both believe capacity is available. The fix made that decision happen in one place before work is selected.

The shape is:

start one pickup decision
count how many jobs this worker already owns
pick up only the remaining open slots
commit

That made capacity a real invariant instead of a best-effort observation.

The Result

After the fix, worker-loop warnings disappeared:

Pre-fix 24h:
179 worker-loop warning/error logs

Post-fix 24h:
0 worker-loop warning/error logs
0 lost-ownership warnings

At the time of the check, worker health was clean:

ready backlog: 0
expired ownership: 0
sleeping work due now: 0
stale running: 0
active waits: 0
due wait timeouts: 0
waiting locks: 0
deadlocks: 0

That is the result we wanted: less database work, fewer worker warnings, and no stuck work.

Separating Worker Health From Job Failures

One thing did not improve: the post-fix 24-hour window had more terminal job failures.

pre-fix: 5,350 completed, 1 failed
post-fix: 8,296 completed, 15 failed

That looked suspicious until we inspected the stored failure reasons.

The failures were not worker-loop failures:

Count	Work type	Cause
11	Message delivery work	Provider server errors while starting the session loop
4	Scheduled and delivery work	Application schema conversion bug

This distinction matters. A durable job can fail because the work inside it fails. That does not mean the worker loop failed.

The worker fix reduced database pressure and removed worker-loop warnings. It did not make provider errors disappear, and it did not fix every application schema bug. Keeping that boundary clear kept us from learning the wrong lesson.

How We Verified The Fix

We used three layers of proof.

First, tests covered the worker invariants:

idle queues poll once per interval, not once per slot
one busy queue does not block another selected queue
a saturated worker respects its slot limit
free slots refill without waiting for the full poll interval
concurrent work pickup from the same worker respects worker concurrency

Second, the load harness measured database pressure directly:

active jobs
running runs
lock waits
database backends
pool waiting
database scan stats
cleanup cost

Third, production checks compared clean pre-fix and post-fix windows while excluding rollout overlap. The 6-hour window showed the mechanism sharply; the 24-hour window showed the same pattern over a longer period.

Better Breadcrumbs For The Next Incident

The follow-up observability work was smaller, but important.

The database already stored the exact terminal failure. CloudWatch needed enough context to find that failure quickly.

The logging boundary now works like this:

CloudWatch = breadcrumb trail
stored failure reason = source of truth

Failure logs carry the execution ID, trace ID, attempt number, and structured error fields. The log points to the right execution. The database preserves the exact failure.

The General Rule

The lesson is not “poll less.” That is too shallow.

The real lesson is:

Do not duplicate control loops to express concurrency.

Concurrency is capacity. Polling is discovery.

When every slot asks the database separately, the database becomes the place where the worker figures out its own capacity. The system gets louder, harder to reason about, and easier to overload.

For the durable worker, the cleaner shape was one loop watching active work and open slots. In other parts of the system, the same pattern shows up as one owner for tenancy, one owner for provider payload validation, or one owner for session state.

The durable worker became quieter because the code stopped asking the database to coordinate something the worker could already know.