Topic 12 · Production systems
Backend & distributed systems
The plumbing that keeps thousands of agent runs moving and correct.
An agent platform is a distributed system first and an AI system second. Requests come in through a gateway, a control plane decides what runs where, workers in an execution plane carry out agent loops and tools, and an inference plane turns prompts into tokens.
Between them sit queues, leases and event streams, and the classic problems come along: work that runs twice, work that never finishes, overloaded dependencies. Idempotency, backpressure and durable state are what keep it honest.
In this topic
37
terms, each defined below
Three planes of an agent platform
Requests flow from the gateway through the control plane into a queue, are picked up by workers, and call out to the inference plane for tokens. Raise the arrival rate past what the workers can handle and switch on backpressure to see the difference between a queue that grows forever and one that pushes back.
Edge
API gateway
auth · rate limits
Control plane
Scheduler
admission · leases
Task queue0 / 6
Execution plane
0 runs completed
Inference plane
Model gateway
routing · fallbacks
GPU engines
batching · KV cache
Queue depth over timeover capacity: shedding load
Glossary
37 terms
Backend engineering
Building the server side of an AI product: APIs, data, queues, workers and integrations with model providers.
Control plane
The part of a system that decides and manages (configuration, scheduling, placement, policy) rather than doing the work itself.
Execution plane
Where agent work actually runs: the workers that execute agent loops and tools.
Inference plane
The model-serving layer: GPU clusters, inference engines and routers that turn prompts into tokens.
API gateway
The front door for client requests, handling authentication, rate limits, routing and request shaping.
Model gateway
A proxy in front of one or more model providers that unifies their APIs and handles keys, fallbacks, routing, caching and cost tracking.
Service boundaries
The lines between services: what each one owns, its API and its data.
Modular monolith
One deployable application split into strongly separated modules. It gets clear boundaries without the operational cost of microservices.
Microservices
Many small, independently deployed services talking over the network. Independent scaling and teams, at the cost of distributed-systems complexity.
Distributed systems
Systems of several machines cooperating over a network, which must cope with partial failure, latency and the lack of a shared clock.
Distributed orchestration
Coordinating multi-step work across machines, with retries, timeouts and durable state.
Worker processes
Processes that pull work from a queue and execute it, scaled separately from the API.
Worker pools
Groups of workers sized to demand, sometimes specialised by resource such as CPU, GPU or sandbox slots.
Task scheduling
Deciding when and where tasks run, including delays, priorities and recurring jobs.
Task queues
Queues of units of work waiting for workers, which decouple accepting a request from executing it.
Message queues
Infrastructure that holds messages until consumers process them, smoothing bursts between producers and consumers.
Message brokers
Servers that route and store messages between producers and consumers, such as NATS, RabbitMQ or Kafka.
Asynchronous messaging
Sending a message without waiting for the receiver to process it, so services stay decoupled in time.
Event-driven architecture
Services reacting to published events, such as "run completed" or "tool approved", instead of calling each other directly.
Event messaging
Publishing and subscribing to events through a broker, with fan-out to many consumers.
Distributed coordination
Getting nodes to agree on leaders, locks, membership or configuration, often through a consensus system.
Concurrency limits
Caps on how many operations run at once per user, tenant, tool or dependency, protecting the systems downstream.
Work distribution
Spreading tasks across workers fairly and efficiently.
Distributed locks
Locks held across processes or machines so only one actor works on a resource at a time.
Leases
Time-limited locks or ownership grants that expire on their own if the holder dies, so work never stays stuck.
Lease-based ownership
A worker owns a task only while it keeps renewing the lease. If it crashes, the lease lapses and another worker takes over.
Work stealing
Idle workers taking queued tasks from busy ones to even out the load.
Backpressure
Signalling upstream to slow down when downstream is overloaded, instead of letting queues grow without limit.
Idempotency
The property that doing an operation twice has the same effect as doing it once. Essential wherever retries can happen.
Idempotency keys
Unique IDs attached to requests so the server can recognise and skip duplicates of an operation it has already done.
Transactional outbox
Writing a database change and the event announcing it in the same transaction, then publishing from that outbox table, so the data and the events can never disagree.
Compensating actions
Steps that undo or offset an earlier action when a later step fails, as in the saga pattern, since true distributed transactions are rarely available.
Exactly-once execution
The goal that each operation takes effect exactly once. Exactly-once delivery over a network is impossible, so systems combine at-least-once delivery with idempotency to get the same effect.
At-least-once execution
A guarantee that an operation will run, possibly more than once. Safe only for idempotent operations.
Fault tolerance
Continuing to work correctly when components fail, through redundancy, retries and isolation.
Distributed recovery
Bringing a distributed system back to a consistent state after a failure, for example by reassigning leases and replaying logs.
Resource allocation
Assigning CPU, memory, GPUs and sandbox slots to tasks according to their needs and the policy.