Topic 18 · Training, tools & research
Advanced systems research questions
Open problems where agents and inference infrastructure meet.
Today agent runtimes and inference engines mostly treat each other as black boxes. The runtime sends independent requests; the engine schedules them without knowing that ten of them belong to the same task, share a long prefix, and that one sits on the critical path.
Many open questions come from closing that gap: scheduling agents and inference together, routing by context and cost, and measuring success in tasks completed rather than tokens served. A useful lens is to think of the agent runtime as an operating system.
In this topic
22
terms, each defined below
The agent runtime as an operating system
Classic operating-system ideas map surprisingly well onto agent infrastructure. Select a row to see the analogy and the research question it raises.
Scheduler≈Orchestrator + inference scheduler
Decides who runs next, today in two layers that cannot see each other.
Research question
Can agent orchestration and model serving be scheduled jointly, and fairly?
Glossary
22 terms
Inference-aware agent scheduling
Scheduling agent steps with knowledge of the inference layer, for example grouping calls that share a prefix, or holding back low-priority agents when GPUs are saturated.
Agent-aware inference serving
Inference engines that understand agent workloads: keeping an agent's KV cache warm between steps, prioritising calls on a task's critical path, anticipating the next call.
Joint agent/inference scheduling
Optimising agent orchestration and model serving together instead of treating the inference API as a black box.
GPU-aware orchestration
Orchestration that considers GPU locality, memory and load when placing agent work and model calls.
Multi-tenant agent infrastructure
Shared platforms running many organisations' agents with isolation, fairness, quotas and per-tenant costs.
Bursty agent workloads
Agents produce sudden bursts of model and tool calls, from fan-out and parallel sub-agents, unlike the steadier traffic of chat.
Agent workload characterization
Measuring what agent workloads really look like: number of calls, prompt reuse, tool latency, context growth and branching.
Agent concurrency optimization
Running independent agent steps and tool calls in parallel, and tuning how many run at once to maximise throughput without overload.
Task-throughput optimization
Optimising completed tasks per unit of time or money, rather than tokens per second, as the real objective.
Cross-layer optimization
Coordinated optimisations across the app, agent runtime, inference engine and hardware that no single layer can achieve alone.
Scheduling fairness
Making sure tenants and agents get fair shares of capacity, so one heavy user's bursts cannot starve everyone else.
Cost-aware model routing
Sending each request to the cheapest model likely to handle it well, and escalating to larger models only when needed.
Model placement
Deciding which models run on which GPUs, and how many replicas each gets, given demand and memory.
Accelerator placement
Matching workloads to accelerator types, such as H100s, consumer GPUs or TPUs, by performance per unit of cost.
Resource-aware scheduling
Scheduling with explicit knowledge of what each task needs: memory, KV-cache size, GPU type and sandbox capacity.
Context-aware routing
Routing requests by what is in their context, such as cached prefixes, length or modality, to the best-suited backend.
Agent execution profiling
Profiling whole agent runs to see how time splits between model calls, tools, queueing and overhead.
Dynamic admission control
Adjusting admission thresholds in real time from load signals, such as accepting fewer new agent runs while KV-cache utilisation is high.
Compute resource contention
Workloads competing for the same GPUs, memory bandwidth or CPUs and slowing each other down.
Inference cost optimization
Lowering the cost per token or per task through batching, caching, quantization, routing and right-sized hardware.
Distributed agent benchmarking
Benchmarking agent systems at scale on realistic, concurrent, multi-tenant workloads instead of single runs.
Agent-runtime operating-system analogy
Viewing the agent runtime as an operating system: the model as the CPU, the context window as RAM, tools as system calls, the scheduler deciding which agent runs, and context compaction as paging.