Topic 17 · Training, tools & research
Infrastructure & development tools
The tools a modern agent and inference stack is actually built with.
No single product covers the whole path from a user's click to a GPU kernel. A real system strings together a web framework, a database and a message broker, a workflow engine, an agent framework, an inference engine, and the observability tools to watch it all.
The map below groups the tools in this guide by the layer they live in. Many are interchangeable within a layer, and most of the hard engineering happens at the seams between layers.
In this topic
46
terms, each defined below
Architecture concepts in the atlas
The stack, layer by layer
From the user-facing app at the top down to GPU kernels at the bottom. Select any tool to read what it does.
Apps & APIs
Agent frameworks
Data & messaging
Inference engines
Models & training
Kernels
Infrastructure & delivery
Observability
Testing
These run alongside every layer on the left.
Inference engines
vLLM
A popular open-source LLM inference engine that introduced PagedAttention, with continuous batching, prefix caching and an OpenAI-compatible server.
Glossary
46 terms
ASP.NET Core
Microsoft's cross-platform, high-performance framework for building web APIs and services in C#.
C# / .NET
Microsoft's statically typed language and the runtime and libraries around it, a common choice for enterprise backends.
Entity Framework Core
The .NET object-relational mapper for working with databases through C# classes and LINQ queries.
PostgreSQL
A widely used open-source relational database known for correctness, extensions and JSON support. With the pgvector extension it also stores embeddings.
JSONB
PostgreSQL's binary JSON column type, which can be indexed and queried. Handy for flexible agent state and tool payloads.
Redis
An in-memory data store used for caching, rate limiting, locks, queues and pub/sub.
NATS JetStream
The persistence layer of the NATS messaging system, adding durable streams, replay and acknowledged consumers.
Temporal
A durable execution platform. Workflows are written as ordinary code, their progress is recorded, and they survive crashes by resuming through deterministic replay.
Docker
Tooling to build and run containers, packaging an application together with its dependencies.
Docker Compose
A tool for defining and running multi-container setups from one YAML file, mostly for local development.
Kubernetes
The standard container orchestrator for deploying, scaling and healing services across a cluster, GPU workloads included.
Terraform
An infrastructure-as-code tool that declares cloud resources in configuration files and applies changes reproducibly.
GitHub Actions
GitHub's CI/CD service for running builds, tests and deployments on every change.
OpenTelemetry
The vendor-neutral standard and SDKs for producing traces, metrics and logs, with emerging conventions for LLM and agent spans.
Prometheus
A time-series database and monitoring system that scrapes metrics and evaluates alert rules.
Grafana
Dashboards and visualisation across metrics, logs and traces.
Grafana Tempo
Grafana's distributed tracing backend for storing and querying traces at scale.
Grafana Loki
Grafana's log aggregation system. It indexes labels rather than full text, which keeps it cheap to run.
NVIDIA DCGM Exporter
Exposes NVIDIA Data Center GPU Manager metrics, such as utilisation, memory, temperature and errors, to Prometheus.
xUnit
A popular unit-testing framework for .NET.
Testcontainers
A library that starts real dependencies, such as databases and brokers, in throwaway containers during tests.
k6
A load-testing tool whose scenarios are written in JavaScript, good for measuring API capacity and latency percentiles.
Next.js
A React framework for web apps with server rendering, static export and file-based routing. This site is built with it.
TypeScript
JavaScript with static types, the default for modern web front ends and many agent frameworks.
shadcn/ui
A collection of copy-in React components built on headless primitives and Tailwind CSS. This site uses it on top of Base UI.
Python
The main language of machine-learning research, training and most inference tooling.
vLLM
A popular open-source LLM inference engine that introduced PagedAttention, with continuous batching, prefix caching and an OpenAI-compatible server.
SGLang
An open-source serving engine with RadixAttention for automatic prefix sharing, fast structured outputs and strong support for large MoE models.
llama.cpp
A C/C++ inference engine for running quantized models (GGUF files) on CPUs, Apple Silicon and consumer GPUs.
TensorRT-LLM
NVIDIA's library for heavily optimised LLM inference on NVIDIA GPUs, with compiled kernels, FP8 and FP4, and in-flight batching.
llm-d
An open-source, Kubernetes-native distributed inference stack built around vLLM, adding cache-aware routing and prefill/decode disaggregation.
Microsoft Agent Framework
Microsoft's open-source SDK for building agents and multi-agent workflows in .NET and Python, the successor to Semantic Kernel and AutoGen.
LangGraph
A library from the LangChain team for building agents as stateful graphs, with persistence, checkpoints and human-in-the-loop steps.
CrewAI
A Python framework for orchestrating teams of role-based agents.
Pydantic AI
A Python agent framework from the Pydantic team, built around type-safe tool definitions and validated structured outputs.
Google ADK
Google's Agent Development Kit, an open-source framework for building, evaluating and deploying agents, integrated with Gemini and Vertex AI.
Mastra
A TypeScript framework for building agents, workflows and retrieval pipelines.
CopilotKit
CopilotKit is an open-source toolkit for putting agents inside React apps, with in-app chat, shared state and generative UI. Its team created the AG-UI protocol.
OpenClaw
An open-source, self-hosted personal AI agent that runs on your own machine and is driven through chat apps, extended with tools and skills.
Hermes Agent
An open-source agent from Nous Research, built around its Hermes models, with persistent memory, skills and tool use.
Megatron-LM
NVIDIA's library for training very large transformers with tensor, pipeline, sequence and expert parallelism.
Qwen
Alibaba's family of open-weight models, dense and MoE, covering text, code and vision. Many of its releases are in the gallery.
Llama
Meta's open-weight model family, whose 2023 release kick-started the open-weight ecosystem.
DeepSeek
The Chinese lab behind DeepSeek-V3 and R1, known for multi-head latent attention, fine-grained MoE and very efficient training.
CUDA
NVIDIA's GPU programming platform, the foundation under nearly every training and inference stack.
Triton
The Python-like GPU kernel language and compiler used for custom fused kernels in PyTorch and inference engines (not to be confused with NVIDIA's Triton Inference Server).