Robustness Reprogrammingfor Representation Learning
Can a trained network become robust without changing a single weight? Replace the least-squares estimator hidden in every linear layer with a robust one, and it can.
Every linear layer is a least-squares estimator
A neuron, a convolution kernel or an attention projection computes \(z = \sum_{d} a_d x_d\). That output is the exact minimiser of a squared loss over the per-input contributions \(a_d x_d\), so an attacker who pushes a few contributions far enough can drag it anywhere. Robust statistics has a classic remedy: penalise the absolute residual instead.
The standard layer. With equal weights, a squared loss gives every input unbounded influence on the output.
The same weights \(a_d\) are reused. Outlying contributions are down-weighted, and the loss provably decreases at every step.
Watch NRPM cut the wires an attacker pulls on
Below are two copies of the same neuron with ten inputs. Each edge carries one contribution \(a_d x_d\) into the output, and its thickness is the weight the estimator gives it. LPM weighs every edge equally. NRPM re-weights each edge by how far its contribution sits from the current estimate. Launch the attack and see which output holds.
LPM, the original layer
NRPM, the reprogrammed layer
Influence of a single input
Drag across the chart to move one contribution away from the others. LPM's output follows it in a straight line, without limit. NRPM's response is bounded, so a far outlier barely moves it. This is the influence function of Theorem 3.2.
Choose how much you pay for robustness
Each reprogrammed layer outputs \(\lambda\, z_{\text{LPM}} + (1-\lambda)\, z_{\text{NRPM}}\), and \(\lambda = 1\) is the original network. All three paradigms start from the released MNIST MLP with every λ at 0.5. Paradigm 1 stops there. Paradigms 2 and 3 are fine-tuned side by side with FGSM (ε = 0.3) for five epochs. What you see is a replay of that real training run.
Recorded by scripts/export_paradigms.py. Paradigm 2 finds the paper's pattern on its own: keep about half of LPM in the first layer and switch the deeper layers almost entirely to NRPM. Paradigm 3 also moves each layer's weights by 3–6%, which buys back clean accuracy and the most robustness.
Twelve digits, two networks, one set of weights
Every tile is attacked twice with white-box FGSM: once against the original model (left image) and once against the reprogrammed model (right image), each with its own gradients. Sweep the budget and watch where the predictions break. Click a tile to look inside the network.
Inside the network
The heat maps are the hidden embeddings of the selected digit, and the percentage under each is the relative shift ‖z − z′‖ / ‖z‖. In the original network the perturbation snowballs from layer to layer. The reprogrammed network keeps every layer closer to its clean values.
Loading attack data…
The same recipe scales to ResNets
The paper applies robustness reprogramming to MLPs, LeNet and ResNet-10/18/34 on MNIST, SVHN, CIFAR-10 and ImageNet10, under white-box, black-box and certified evaluation. Below, on CIFAR-10 with ResNet-18, adversarially fine-tuned Paradigm 3 is compared against DYNAT, the strongest adversarial-training baseline on average.
CIFAR-10 · ResNet-18 · robust accuracy (%)
Robust accuracy on CIFAR-10 with ResNet-18, up from 45.1% for DYNAT and 42.1% for PGD-AT.
Averaged over PGD, FGSM, C&W, AutoAttack, DeepFool and SPSA. The best baseline averages 52.0%.
Paradigm 1 only swaps the estimator. It needs no training and complements any existing defense.
Citation
@inproceedings{hou2025robustness,
title = {Robustness Reprogramming for Representation Learning},
author = {Hou, Zhichao and Torkamani, MohamadAli and Krim, Hamid and Liu, Xiaorui},
booktitle = {The Thirteenth International Conference on Learning Representations (ICLR)},
year = {2025},
note = {Spotlight},
url = {https://arxiv.org/abs/2410.04577}
}
Abstract
This work tackles an intriguing and fundamental open challenge in representation learning: given a well-trained deep learning model, can it be reprogrammed to enhance its robustness against adversarial or noisy input perturbations without altering its parameters? To explore this, we revisit the core feature transformation mechanism in representation learning and propose a novel non-linear robust pattern matching technique as a robust alternative. Furthermore, we introduce three model reprogramming paradigms to offer flexible control of robustness under different efficiency requirements. Comprehensive experiments and ablation studies across diverse learning models, ranging from basic linear models and MLPs to shallow and modern deep ConvNets, demonstrate the effectiveness of our approaches. This work not only opens a promising and orthogonal direction for improving adversarial defenses in deep learning beyond existing methods but also provides new insights into designing more resilient AI systems with robust statistics.
Zhichao Hou and Xiaorui Liu are supported by the NSF National AI Research Resource Pilot Award, an Amazon Research Award, the NCSU Data Science Academy Seed Grant Award, and the NCSU Faculty Research and Professional Development Award.