Mastering torch.profiler to Squeeze More Performance from PyTorch

What you cannot profile, you cannot optimize. Whether you are trying to squeeze more tokens per second out of a Large Language Model (LLM) or shave milliseconds off inference, the path eventually leads to profiling.

Most tutorials assume you can already read the dense walls of colored rectangles in a trace. This guide breaks down how to use torch.profiler to decode those artifacts and turn them into actionable optimization insights.

Decoding the two artifacts of torch.profiler

When you run a profile on an NVIDIA A100-SXM4-80GB GPU, torch.profiler hands back two distinct artifacts that answer different engineering questions:

1. The Profiler Table: This provides a statistical summary of your algorithm. It answers "What is taking the most time?" and helps you identify hotspots—events that take the most amount of time or are triggered too frequently.
2. The Profiler Trace: This provides a temporal execution view. It answers "When and Why an operation happened," depicting the activities taking place on the CPU and the GPU. This is essential for investigating kernel launches, launch delays, or any overlap between CPU and GPU activities.

Setting up a baseline with matrix multiplication

To understand the mechanics, we start with the most fundamental operation in deep neural networks: a matrix multiplication followed by a bias add. This mimics how weights and biases interact in a neuron and serves as a perfect baseline for understanding how profiling paves the way for compilation later.

Using torch.profiler.profile with both ProfilerActivity.CPU and ProfilerActivity.CUDA enabled, you can capture the chain of events from a Python call all the way down to a CUDA kernel. By using torch.profiler.record_function("matmul_add"), you can annotate your algorithm to make it easily navigable within the dense traces.

By mastering these two artifacts, you can begin to move from a beginner's view of the traces to a professional understanding of how your code actually executes on the hardware, setting the stage for more complex optimizations like torch.compile.

Understanding the temporal execution view is the first step toward building a deep, technical intuition for how your code actually runs on the hardware, which enables more advanced performance tuning in the future.

Source: Profiling in PyTorch (Part 1): A Beginner's Guide to torch.profiler
Domain: huggingface.co