AgileOS Virtualizes CUDA to Protect GPU Services Without Kernel Hacks

Modern GPU applications do a lot more than crunch kernels—they talk to storage, network devices, vendor libraries, and GPU-resident services. CUDA gives each application direct ownership of its context, device pointers, and kernel launches, leaving those services to hack together their own isolation. That’s a security gap the size of a memory-mapped I/O region.

AgileOS, a new GPU operating‑system layer from the paper “AgileOS: A GPU Operating System Layer for Protected CUDA Services” (arXiv:2606.06697), fixes this by virtualizing CUDA at the library boundary. Applications link against shim versions of the CUDA Runtime, Driver, and selected libraries. A trusted runtime worker owns the real CUDA context and mediates every operation. No kernel patches, no driver modifications—just smart interception.

CUDA’s Security Gap for GPU Services

CUDA’s programming model treats each application as the sole owner of its context and resources. That works fine for isolated compute kernels, but GPU-resident services—vendor libraries, caching layers, networked inference servers—expose service metadata, device queues, and MMIO regions directly to untrusted kernels. Every ad‑hoc protection scheme is brittle and incompatible.

AgileOS addresses exactly this. It separates user allocations from protected module and MMIO ranges using a custom GPU memory manager. Pointer validation and memory access guards are injected at the PTX level, so a rogue kernel can’t read or write service state even if it knows the address. The paper calls this “protected module/MMIO ranges” enforced via PTX injection—hardware‑level enforcement without touching the GPU’s firmware.

Library‑Level Virtualization with PTX Guards

The prototype has client‑side interceptors, worker‑side CUDA handlers, virtualized CUDA object tables, and a set of protected AgileOS modules. The architecture is modular: you can plug in different services without rewriting the isolation layer. The GPU memory manager is the key component—it tracks which addresses belong to user allocations and which belong to protected services, and the PTX‑level guard rejects any kernel access that violates the boundary.

This is not a hypervisor or a kernel module. It’s a userspace layer that rewires CUDA calls through a trusted worker. The worker owns the real CUDA context and validates every operation before forwarding it to the driver. For an engineer who’s ever tried to run a multi‑tenant GPU service, this feels like the right level of abstraction—minimal overhead, strong isolation, no fork of the driver stack.

Supporting cuFFT and PyTorch Out of the Box

AgileOS already includes adapters for cuFFT and PyTorch. That means you can run a PyTorch inference service with its internal state protected from the user’s input‑processing kernels. The paper shows the architecture supports “a range of protected services and existing libraries,” so expect more adapters as the prototype matures.

What this enables is straightforward: secure multi‑tenant GPU environments where untrusted application kernels cannot corrupt or exfiltrate service data. No more praying that your ad‑hoc sandbox is airtight. AgileOS virtualizes CUDA, and that’s the kind of OS‑level thinking that GPUs have needed for years.

Source: AgileOS: A GPU Operating System Layer for Protected CUDA Services
Domain: arxiv.org