Localized Gaussians as Self-Attention Weights for Point Clouds Correspondence
Loading...
Date
2024
Journal Title
Journal ISSN
Volume Title
Publisher
The Eurographics Association
Abstract
Current data-driven methodologies for point cloud matching demand extensive training time and computational resources, presenting significant challenges for model deployment and application. In the point cloud matching task, recent advancements with an encoder-only Transformer architecture have revealed the emergence of semantically meaningful patterns in the attention heads, particularly resembling Gaussian functions centered on each point of the input shape. In this work, we further investigate this phenomenon by integrating these patterns as fixed attention weights within the attention heads of the Transformer architecture. We evaluate two variants: one utilizing predetermined variance values for the Gaussians, and another where the variance values are treated as learnable parameters. Additionally we analyze the performances on noisy data and explore a possible way to improve robustness to noise. Our findings demonstrate that fixing the attention weights not only accelerates the training process but also enhances the stability of the optimization. Furthermore, we conducted an ablation study to identify the specific layers where the infused information is most impactful and to understand the reliance of the network on this information.
Description
CCS Concepts: Computing methodologies → Machine learning; Shape analysis; Theory of computation → Computational geometry
@inproceedings{10.2312:stag.20241345,
booktitle = {Smart Tools and Applications in Graphics - Eurographics Italian Chapter Conference},
editor = {Caputo, Ariel and Garro, Valeria and Giachetti, Andrea and Castellani, Umberto and Dulecha, Tinsae Gebrechristos},
title = {{Localized Gaussians as Self-Attention Weights for Point Clouds Correspondence}},
author = {Riva, Alessandro and Raganato, Alessandro and Melzi, Simone},
year = {2024},
publisher = {The Eurographics Association},
ISSN = {2617-4855},
ISBN = {978-3-03868-265-3},
DOI = {10.2312/stag.20241345}
}