PAELLA: Parameter-Efficient Lightweight Language-Agnostic Captioning Model
Research output: Chapter in Book/Report/Conference proceeding › Article in proceedings › Research › peer-review
Standard
PAELLA : Parameter-Efficient Lightweight Language-Agnostic Captioning Model. / Ramos, Rita; Bugliarello, Emanuele; Martins, Bruno; Elliott, Desmond.
Findings of the Association for Computational Linguistics: NAACL 2024. ed. / Kevin Duh; Helena Gomez; Steven Bethard. Association for Computational Linguistics (ACL), 2024. p. 3549-3564.Research output: Chapter in Book/Report/Conference proceeding › Article in proceedings › Research › peer-review
Harvard
APA
Vancouver
Author
Bibtex
}
RIS
TY - GEN
T1 - PAELLA
T2 - 2024 Findings of the Association for Computational Linguistics: NAACL 2024
AU - Ramos, Rita
AU - Bugliarello, Emanuele
AU - Martins, Bruno
AU - Elliott, Desmond
N1 - Publisher Copyright: © 2024 Association for Computational Linguistics.
PY - 2024
Y1 - 2024
N2 - We introduce PAELLA, a Parameter-Efficient Lightweight Language-Agnostic image captioning model designed to be both parameter and data-efficient using retrieval augmentation. The model is trained by learning a small mapping network with 34M parameters between a pre-trained visual model and a multilingual language model that is conditioned on two types of input: (i) the image itself, and (ii) a set of retrieved captions in the target language. The retrieved examples play a key role in guiding the model to generate captions across languages. Through retrieval, the model can be lightweight in terms of the number of trainable parameters, which only exist in its mapping network, and also in the amount of multilingual training data that is required. Experiments on the XM3600 dataset, featuring 36 languages, show that PAELLA can outperform or compete against some models with 3-77× more learned parameters and 35-863× more data, particularly in low-resource languages. We also find that PAELLA can be trained on only monolingual data and still show strong zero-shot abilities in other languages.
AB - We introduce PAELLA, a Parameter-Efficient Lightweight Language-Agnostic image captioning model designed to be both parameter and data-efficient using retrieval augmentation. The model is trained by learning a small mapping network with 34M parameters between a pre-trained visual model and a multilingual language model that is conditioned on two types of input: (i) the image itself, and (ii) a set of retrieved captions in the target language. The retrieved examples play a key role in guiding the model to generate captions across languages. Through retrieval, the model can be lightweight in terms of the number of trainable parameters, which only exist in its mapping network, and also in the amount of multilingual training data that is required. Experiments on the XM3600 dataset, featuring 36 languages, show that PAELLA can outperform or compete against some models with 3-77× more learned parameters and 35-863× more data, particularly in low-resource languages. We also find that PAELLA can be trained on only monolingual data and still show strong zero-shot abilities in other languages.
M3 - Article in proceedings
AN - SCOPUS:85197894973
SP - 3549
EP - 3564
BT - Findings of the Association for Computational Linguistics
A2 - Duh, Kevin
A2 - Gomez, Helena
A2 - Bethard, Steven
PB - Association for Computational Linguistics (ACL)
Y2 - 16 June 2024 through 21 June 2024
ER -
ID: 398633242