# 18. Advanced systems research questions

> Open problems where agents and inference infrastructure meet.

Track: Training, tools & research. 22 terms.

Today agent runtimes and inference engines mostly treat each other as black boxes. The runtime sends independent requests; the engine schedules them without knowing that ten of them belong to the same task, share a long prefix, and that one sits on the critical path.

Many open questions come from closing that gap: scheduling agents and inference together, routing by context and cost, and measuring success in tasks completed rather than tokens served. A useful lens is to think of the agent runtime as an operating system.

**Interactive figure on the web page: The agent runtime as an operating system.** Classic operating-system ideas map surprisingly well onto agent infrastructure. Select a row to see the analogy and the research question it raises.

## Terms

### Inference-aware agent scheduling

Scheduling agent steps with knowledge of the inference layer, for example grouping calls that share a prefix, or holding back low-priority agents when GPUs are saturated.

### Agent-aware inference serving

Inference engines that understand agent workloads: keeping an agent's KV cache warm between steps, prioritising calls on a task's critical path, anticipating the next call.

### Joint agent/inference scheduling

Optimising agent orchestration and model serving together instead of treating the inference API as a black box.

### GPU-aware orchestration

Orchestration that considers GPU locality, memory and load when placing agent work and model calls.

### Multi-tenant agent infrastructure

Shared platforms running many organisations' agents with isolation, fairness, quotas and per-tenant costs.

### Bursty agent workloads

Agents produce sudden bursts of model and tool calls, from fan-out and parallel sub-agents, unlike the steadier traffic of chat.

### Agent workload characterization

Measuring what agent workloads really look like: number of calls, prompt reuse, tool latency, context growth and branching.

### Agent concurrency optimization

Running independent agent steps and tool calls in parallel, and tuning how many run at once to maximise throughput without overload.

### Task-throughput optimization

Optimising completed tasks per unit of time or money, rather than tokens per second, as the real objective.

### Cross-layer optimization

Coordinated optimisations across the app, agent runtime, inference engine and hardware that no single layer can achieve alone.

### Scheduling fairness

Making sure tenants and agents get fair shares of capacity, so one heavy user's bursts cannot starve everyone else.

### Cost-aware model routing

Sending each request to the cheapest model likely to handle it well, and escalating to larger models only when needed.

### Model placement

Deciding which models run on which GPUs, and how many replicas each gets, given demand and memory.

### Accelerator placement

Matching workloads to accelerator types, such as H100s, consumer GPUs or TPUs, by performance per unit of cost.

### Resource-aware scheduling

Scheduling with explicit knowledge of what each task needs: memory, KV-cache size, GPU type and sandbox capacity.

### Context-aware routing

Routing requests by what is in their context, such as cached prefixes, length or modality, to the best-suited backend.

See: [related](https://llm-atlas-7or.pages.dev/learn/distributed/#cache-aware-routing)

### Agent execution profiling

Profiling whole agent runs to see how time splits between model calls, tools, queueing and overhead.

See: [related](https://llm-atlas-7or.pages.dev/learn/observability/)

### Dynamic admission control

Adjusting admission thresholds in real time from load signals, such as accepting fewer new agent runs while KV-cache utilisation is high.

### Compute resource contention

Workloads competing for the same GPUs, memory bandwidth or CPUs and slowing each other down.

### Inference cost optimization

Lowering the cost per token or per task through batching, caching, quantization, routing and right-sized hardware.

### Distributed agent benchmarking

Benchmarking agent systems at scale on realistic, concurrent, multi-tenant workloads instead of single runs.

### Agent-runtime operating-system analogy

Viewing the agent runtime as an operating system: the model as the CPU, the context window as RAM, tools as system calls, the scheduler deciding which agent runs, and context compaction as paging.

Web page: https://llm-atlas-7or.pages.dev/learn/research/

_Definitions written for LLM Atlas._