TIGER Attack Inverts Transformer Gradients via Embedding-Subspace Optimization

TIGER turns one of gradient inversion's biggest weaknesses into a strength: instead of groping through a space of dummy inputs or testing discrete tokens one by one, it directly optimizes token embeddings to lie in a subspace defined by the low-rank attention gradients.

Why Prior Attacks Stumble on Transformers

Existing gradient inversion attacks on transformers fall into two camps, both flawed. Dummy-input optimization tries to match the full gradient of a fake input to the real client update, but that becomes costly and unstable for modern transformer architectures. The other approach exploits the low rank of attention gradients to identify a subspace containing the true layer embeddings, then runs a discrete membership test over candidate tokens. That token test is brittle under numerical noise from quantization or differential privacy (DP), and it scales poorly for encoder-only models with non-causal attention.

TIGER's Differentiable Subspace Objective

TIGER replaces the discrete token search with a differentiable objective that continuously minimizes the distance between token embeddings and the identified subspace. No more enumerating candidates or matching full gradients. On encoder-only models, the paper reports substantial improvements in both reconstruction quality and runtime compared to prior subspace-based attacks. For decoder models, TIGER is more robust than those earlier methods, handling numerical noise that would break the discrete approach.

First Reconstruction Under Differential Privacy

The attack's resilience to noise opens a new frontier: DP-defended federated learning. Prior subspace attacks failed under even modest DP noise because the discrete membership test could not tolerate the perturbation. TIGER's continuous optimization adapts, and the authors claim the first successful reconstructions of client inputs from transformer gradients in DP-defended settings. That directly challenges the assumption that gradient perturbations alone are sufficient to protect privacy when transformers are used in federated learning.

Gradient inversion is not new, but TIGER shifts the practical threat model: any federated learning system using transformers, even with DP or quantization, should assume that shared gradient updates can leak input data.

Source: TIGER: Inverting Transformer Gradients via Embedding-Subspace Distance Optimization
Domain: arxiv.org