Talking to DINO

Bridging Self-Supervised Vision Backbones with Language for Open-Vocabulary Segmentation

Luca Barsellotti*1, Lorenzo Bianchi*2,3, Nicola Messina2, Fabio Carrara2, Marcella Cornia1, Lorenzo Baraldi1, Fabrizio Falchi2 Rita Cucchiara1
1University of Modena and Reggio Emilia, Italy 2ISTI CNR 3University of Pisa, Italy
arXiv Code Models
Talk2DINO pipeline

Abstract

Open-Vocabulary Segmentation (OVS) aims at segmenting images from free-form textual concepts without predefined training classes. While existing vision-language models such as CLIP can generate segmentation masks by leveraging coarse spatial information from Vision Transformers, they face challenges in spatial localization due to their global alignment of image and text features. Conversely, self-supervised visual models like DINO excel in fine-grained visual encoding but lack integration with language.

To bridge this gap, we present Talk2DINO, a novel hybrid approach that combines the spatial accuracy of DINOv2 with the language understanding of CLIP. Our approach aligns the textual embeddings of CLIP to the patch-level features of DINOv2 through a learned mapping function without the need to fine-tune the underlying backbones. At training time, we exploit the attention maps of DINOv2 to selectively align local visual patches with textual embeddings.

We show that the powerful semantic and localization abilities of Talk2DINO can enhance the segmentation process, resulting in more natural and less noisy segmentations, and that our approach can also effectively distinguish foreground objects from the background. Experimental results demonstrate that Talk2DINO achieves state-of-the-art performance across several unsupervised OVS benchmarks.

Examples

Results examples

Qualitative results of Talk2DINO in comparison with FreeDA [3], ProxyCLIP [23], and CLIP-DINOiser [47].

Results

Results examples

Comparison with unsupervised OVS models on Pascal VOC [15], Pascal Context [30], COCO Stuff [7], Cityscapes [11], and ADE20K [56, 57]. For each method, we specify the visual backbone used, along with whether it is frozen or fine-tuned.

BibTeX

@misc{barsellotti2024talkingdinobridgingselfsupervised,
  title={Talking to DINO: Bridging Self-Supervised Vision Backbones with Language for Open-Vocabulary Segmentation}, 
  author={Luca Barsellotti and Lorenzo Bianchi and Nicola Messina and Fabio Carrara and Marcella Cornia and Lorenzo Baraldi and Fabrizio Falchi and Rita Cucchiara},
  year={2024},
  eprint={2411.19331},
  archivePrefix={arXiv},
  primaryClass={cs.CV},
  url={https://arxiv.org/abs/2411.19331}, 
}

Acknowledgements

MUCES Project Logo This work has received financial support by the European Union — Next Generation EU, Mission 4 Component 1 CUP B53D23026090001 and E53D23016290001 (a MUltimedia platform for Content Enrichment and Search in audiovisual archives — MUCES PRIN 2022 PNRR P2022BW7CW).

SUN Project Logo This work has received financial support by the Horizon Europe Research & Innovation Programme under Grant agreement N. 101092612 (Social and hUman ceNtered XR - SUN project).