We propose PVeRA, a probabilistic adapter that learns a distribution over weight adaptations rather than a fixed one. Built on top of VeRA's frozen random matrices, PVeRA uses a reparameterization trick to sample latent adaptations at each forward pass, adding calibrated uncertainty estimates at virtually no extra parameter cost. On the 19-dataset VTAB-1k benchmark with a DINOv2 backbone, PVeRA outperforms VeRA and six other adapters (avg. 71.4% vs. 69.9%), stays well-calibrated, enables confidence interval estimation, and supports out-of-distribution detection.
PVeRA is integrated into the
🤗 PEFT library.
Install PEFT and use PveraConfig as a drop-in replacement for other adapter configs.
See the full documentation
for all configuration options including pvera_dropout, target_modules,
layers_to_transform, and sample_at_inference.
from transformers import AutoModel
from peft import PveraConfig, get_peft_model
base_model = AutoModel.from_pretrained("facebook/dinov2-base")
config = PveraConfig(r=256, target_modules=["query", "value"], pvera_dropout=0.0, sample_at_inference=False)
model = get_peft_model(base_model, config)We release 19 pretrained PVeRA adapters based on DINOv2-B, one per VTAB-1k dataset. All adapters are available in the 🤗 Hugging Face collection.
from transformers import AutoModel
from peft import PeftModel
# Available datasets: caltech101, cifar, dtd, flowers102, pets, sun397, svhn, camelyon,
# eurosat, resisc45, retinopathy, clevrcount, clevrdist, dmlab,
# dspritesloc, dspritesori, kittidist, smallnorbazi, smallnorbele
dataset = "caltech101"
base_model = AutoModel.from_pretrained("facebook/dinov2-base")
model = PeftModel.from_pretrained(base_model, f"leoflx/pvera_dinov2_b_{dataset}")
model.eval()Let \(\mathbf{x} \in \mathbb{R}^{l \times d}\) be the input to the attention mechanism, with \(l \in \mathbb{N}_+^*\) the sequence length and \(d \in \mathbb{N}_+^*\) the feature space dimensionality. Self-attention is obtained using linear layers with weights \(\mathbf{W}_q, \mathbf{W}_k, \mathbf{W}_v \in \mathbb{R}^{d \times d}\) and biases \(\mathbf{b}_{W_q}, \mathbf{b}_{W_k}, \mathbf{b}_{W_v} \in \mathbb{R}^d\):
LoRA is applied to the query and value branches. Given rank \(r \in \mathbb{N}_+^* < d\) and scaling \(\alpha \in \mathbb{R}_+^*\), its only trainable parameters are \(\mathbf{A} \in \mathbb{R}^{d \times r}\) and \(\mathbf{B} \in \mathbb{R}^{r \times d}\):
VeRA shares frozen random matrices \(\mathbf{A}\) and \(\mathbf{B}\) across all layers. Its only trainable parameters are \(\mathbf{d} \in \mathbb{R}^r\) and \(\mathbf{b} \in \mathbb{R}^d\):
Our proposed adapter, PVeRA, is a probabilistic adaptation of VeRA. \(\mathbf{A}_{\{q,v\}} \in \mathbb{R}^{d \times 2r}\) and \(\mathbf{d}_{\{q,v\}} \in \mathbb{R}^{2r}\) are used to generate \(\boldsymbol{\mu}_{\{q,v\}} \in \mathbb{R}^r\) and \(\boldsymbol{\sigma}_{\{q,v\}} \in \mathbb{R}^r\), representing the mean and standard deviation of a multivariate normal distribution. Using the reparameterization trick, we sample from the learned distribution of the latent space \(\mathbf{z}_{\{q,v\}} \sim \mathcal{N}(\boldsymbol{\mu}_{\{q,v\}}, \boldsymbol{\sigma}^2_{\{q,v\}})\):
We use a KL divergence loss to enforce a standard Normal prior to each PVeRA adapter. With \(\beta \in \mathbb{R}_+^*\) defined as the KL loss scaling factor (defined per dataset using a grid search on the validation loss):
During inference, the adaptation can be performed in a deterministic or probabilistic fashion. For deterministic inference, we use \(\mathbf{z}_{\{q,v\}} = \boldsymbol{\mu}_{\{q,v\}}\). As with VeRA, the weights can then be merged into the original model weights, resulting in no additional inference time:
Alternatively, a probabilistic adaptation can be used such that adaptations are randomly drawn from the learned distribution, enabling Monte Carlo confidence interval estimation. Unless otherwise stated, we use deterministic inference in our experiments.
We benchmark PVeRA against seven adapters on the 19 datasets of VTAB-1k using three sizes of DINOv2 as backbone. Results are average accuracy (%) across three random seeds. Bold indicates the best result. Differences for PVeRA are reported with respect to VeRA.
| Model | Linear | Bottleneck | (IA)3 | AdaptFormer | DoRA | LoRA | VeRA | PVeRA |
|---|---|---|---|---|---|---|---|---|
| DINOv2-S | 53.6 | 51.5 | 64.0 | 60.1 | 66.1 | 66.1 | 65.9 | 67.5+1.6 |
| DINOv2-B | 57.0 | 68.2 | 66.6 | 69.6 | 71.0 | 70.5 | 69.9 | 71.4+1.5 |
| DINOv2-L | 56.6 | 70.2 | 63.9 | 51.3 | 71.2 | 73.1 | 71.9 | 73.3+1.4 |
Results on VTAB-1k across 3 models.
At inference time, PVeRA can perform multiple stochastic forward passes by sampling from the learned adapter distributions. Accumulating the softmax scores across passes yields a Monte Carlo confidence interval for the predicted class.
Input image
0 / 16 passes
@inproceedings{fillioux2025pvera,
title={{PVeRA}: Probabilistic Vector-Based Random Matrix Adaptation},
author={Fillioux, Leo and Ferrante, Enzo and Cournède, Paul-Henry and Vakalopoulou, Maria and Christodoulidis, Stergios},
booktitle={Proceedings of the Winter Conference on Applications of Computer Vision (WACV)},
year={2026}
}|
This template was inspired from
this page
and
this one
.
|