ICLR 2026

Fine-Grained Iterative Adversarial Attacks with Limited Computation Budget

North Carolina State University

Cutting PGD iterations to fit a compute budget weakens the attack. Spiking-PGD recomputes a layer only when its input has changed enough, reuses the cached output otherwise, and keeps the gradient alive with a virtual surrogate gradient.

30%

of the PGD-AT budget for comparable clean and robust accuracy in adversarial training

benchmarks across vision and graphs: CIFAR-10, CIFAR-100, Tiny-ImageNet, Cora, Citeseer

knob, the spiking threshold ρ, sets the trade-off between cost and attack strength

Abstract

This work tackles a critical challenge in AI safety research under limited compute: given a fixed computation budget, how can one maximize the strength of iterative adversarial attacks? Coarsely reducing the number of attack iterations lowers cost but substantially weakens effectiveness. To fulfill the attainable attack efficacy within a constrained budget, we propose a fine-grained control mechanism that selectively recomputes layer activations across both iteration-wise and layer-wise levels. Extensive experiments show that our method consistently outperforms existing baselines at equal cost. Moreover, when integrated into adversarial training, it attains comparable performance with only 30% of the original budget.

Animation

One decision per layer, per iteration

Each PGD step runs the input forward through all 18 layers. Spiking-PGD checks every layer as the forward pass reaches it: if its input changed by at least \(\rho\) since the last step, the layer is recomputed; otherwise the whole layer reuses its cached output \(\hat o_{t-1}\). The decision is made afresh at every iteration, so a layer that was reused can be recomputed at full precision later if its input starts moving again. Watch the iteration × layer mask fill in.

T1of 10 PGD iterations

ρ ×10⁻² 1.0

Vanilla PGD0/ 180

recomputes every layer; 0 of these were layers whose input barely changed

Spiking-PGD0/ 180

0 layers reused instead · – of vanilla's compute

Illustrative scenario: based on the measured Fig. 1 values, with a few layers whose change rises again at later iterations, so they are reused first and then recomputed at full precision (marked ↻). Switch to Measured for the paper's ResNet-18 data, where changes decay steadily. The first iteration is always computed in full.

Method

Spiking Iterative Attack

Iterative attacks are redundant

Across PGD iterations, hidden activations quickly stop changing, and they settle at different rates in different layers. Adversarially trained models change even less. Recomputing every layer at every step wastes most of the budget.

Relative activation change across attack iterations and layers for ResNet-18 on CIFAR-10. — Relative activation change for ResNet-18 on CIFAR-10. **Left:** per-layer curves (light) and layer average (dark) for normal and robust models. **Right:** across layers and iterations.

Budget allocation as a layer-wise mask

Let \(\Delta\in\{0,1\}^{T\times L}\) mark whether layer \(l\) is recomputed at iteration \(t\). Early stopping is the block-structured special case of this problem, so its optimum can never beat the fine-grained one: \(V_{\text{coarse}}\le V_{\text{fine}}\).

\[\max_{\Delta\in\{0,1\}^{T\times L}}\ \mathcal{L}\big(\mathbf{x}_{T+1}(\Delta),y\big)\quad\text{s.t.}\quad\sum_{t=1}^{T}\sum_{l=1}^{L}C_{t,l}\,\delta_{t,l}\le C_{\text{total}}\]

Spiking forward computation

For a linear layer, the new output equals the old output plus the layer applied to the activation residual. A gate with one threshold \(\rho\) fires only when that residual is large, and otherwise reuses \(\hat o_{t-1}\). One cheap rule replaces a search over an exponentially large mask space.

\[\mathcal{S}_\rho(a_t,a_{t-1})=\begin{cases}a_t-a_{t-1}, & \|a_t-a_{t-1}\|/\|a_t\|\ge\rho\\[2pt] \mathbf{0}, & \text{otherwise}\end{cases}\]

Vanilla forward

\[\begin{aligned} o_1 &= \mathcal A(a_1)\\ o_2 &= \mathcal A(a_2-a_1)+o_1\\[-2pt] &\;\;\vdots\\[-2pt] o_t &= \mathcal A(a_t-a_{t-1})+o_{t-1}\end{aligned}\]

Spiking forward

\[\begin{aligned} \hat o_1 &= \mathcal A(a_1)\\ \hat o_2 &= \mathcal A\big(\class{fire}{\mathcal S_\rho(a_2,a_1)}\big)+\hat o_1\\[-2pt] &\;\;\vdots\\[-2pt] \hat o_t &= \mathcal A\big(\class{fire}{\mathcal S_\rho(a_t,a_{t-1})}\big)+\hat o_{t-1}\end{aligned}\]

Per-layer outputs across attack iterations (layer index \((l)\) omitted). Left: vanilla computation written as residual accumulation, using the linearity of \(\mathcal A\). Right: spiking computation passes the residual through the gate \(\mathcal S_\rho\).

Virtual surrogate gradient

Reusing a cached output cuts the autograd graph, and the gradient to the input vanishes. We route the upstream gradient back through the layer's adjoint, \(\partial\mathcal L/\partial a_t=\mathcal A^{\top}(\partial\mathcal L/\partial\hat o)\), using conv2d_input for convolutions and a matrix product for linear layers.

(a) Exact gradient

\[\frac{\partial\mathcal L}{\partial a_t}=\mathcal A^{\top}\Big(\frac{\partial\mathcal L}{\partial\hat o_t}\Big)\]

Layer computed: autograd follows the chain rule.

(b) Vanishing gradient

\[\frac{\partial\mathcal L}{\partial a_t}=\mathbf 0\]

Output reused: the gate blocks the path to \(a_t\).

\[\frac{\partial\mathcal L}{\partial a_t}=\mathcal A^{\top}\Big(\frac{\partial\mathcal L}{\partial\hat o_{t-1}}\Big)\]

Ours: route the upstream gradient through \(\mathcal A^{\top}\).

Results

Stronger at every budget

Accuracy under attack against the compute the attack is allowed to spend; lower means a stronger attack. Baselines spend their budget by running fewer iterations, Spiking-PGD by reusing layers. Drag across the chart to pick a budget.

Vision · ResNet-18

Graph · GCN

Budget

At 15% of full compute

–accuracy points below the best baseline

Accuracy remaining under attack. A shorter bar is a stronger attack.

Curves are read off Figs. 5 and 6 of the paper (accurate to a few tenths of a point). A few high-budget baseline points hidden behind overlapping curves in the figures are left out.

Adversarial training at a fraction of the cost

For adversarial training the threshold follows a schedule, \(\rho(t)=\rho_0\frac{e^{-\lambda t/N}-e^{-\lambda}}{1-e^{-\lambda}}\) with \(\rho_0=0.1\) and \(N=200\) epochs. Early on, ρ is large and the attack reuses almost everything, so perturbations are cheap and weak. As ρ decays, more layers are recomputed and the attack sharpens. Play the 200 epochs on CIFAR-10.

Epoch 0of 200

Attack compute so far–of what PGD-AT spends

Threshold ρ–

This epoch–

Robust acc · ours–

Robust acc · PGD-AT–

Attack mask this epoch (illustrative)

Accuracy and per-epoch attack compute are read off Figs. 7 and 8 of the paper; compute counts full-precision layer evaluations in the 10-step training attack. The mask shows a pattern with the same compute fraction, built from the Fig. 1 statistics.

Citation

BibTeX

@inproceedings{hou2026fine,
  title={Fine-Grained Iterative Adversarial Attacks with Limited Computation Budget},
  author={Hou, Zhichao and Gao, Weizhi and Liu, Xiaorui},
  booktitle={International Conference on Learning Representations},
  volume={2026},
  pages={75235--75250},
  year={2026}
}