A Multilingual Benchmark for Probing Negation-Awareness with Minimal Pairs
Research output: Chapter in Book/Report/Conference proceeding › Article in proceedings › Research › peer-review
Standard
A Multilingual Benchmark for Probing Negation-Awareness with Minimal Pairs. / Hartmann, Mareike; de Lhoneux, Miryam; Hershcovich, Daniel; Kementchedjhieva, Yova Radoslavova; Nielsen, Lukas Christian; Qiu, Chen; Søgaard, Anders.
Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2021. p. 244–257.Research output: Chapter in Book/Report/Conference proceeding › Article in proceedings › Research › peer-review
Harvard
APA
Vancouver
Author
Bibtex
}
RIS
TY - GEN
T1 - A Multilingual Benchmark for Probing Negation-Awareness with Minimal Pairs
AU - Hartmann, Mareike
AU - de Lhoneux, Miryam
AU - Hershcovich, Daniel
AU - Kementchedjhieva, Yova Radoslavova
AU - Nielsen, Lukas Christian
AU - Qiu, Chen
AU - Søgaard, Anders
PY - 2021
Y1 - 2021
N2 - Negation is one of the most fundamental concepts in human cognition and language, and several natural language inference (NLI) probes have been designed to investigate pretrained language models’ ability to detect and reason with negation. However, the existing probing datasets are limited to English only, and do not enable controlled probing of performance in the absence or presence of negation. In response, we present a multilingual (English, Bulgarian, German, French and Chinese) benchmark collection of NLI examples that are grammatical and correctly labeled, as a result of manual inspection and reformulation. We use the benchmark to probe the negation-awareness of multilingual language models and find that models that correctly predict examples with negation cues, often fail to correctly predict their counter-examples without negation cues, even when the cues are irrelevant for semantic inference.
AB - Negation is one of the most fundamental concepts in human cognition and language, and several natural language inference (NLI) probes have been designed to investigate pretrained language models’ ability to detect and reason with negation. However, the existing probing datasets are limited to English only, and do not enable controlled probing of performance in the absence or presence of negation. In response, we present a multilingual (English, Bulgarian, German, French and Chinese) benchmark collection of NLI examples that are grammatical and correctly labeled, as a result of manual inspection and reformulation. We use the benchmark to probe the negation-awareness of multilingual language models and find that models that correctly predict examples with negation cues, often fail to correctly predict their counter-examples without negation cues, even when the cues are irrelevant for semantic inference.
U2 - 10.18653/v1/2021.conll-1.19
DO - 10.18653/v1/2021.conll-1.19
M3 - Article in proceedings
SP - 244
EP - 257
BT - Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing
PB - Association for Computational Linguistics
T2 - 2021 Conference on Empirical Methods in Natural Language Processing
Y2 - 7 November 2021 through 11 November 2021
ER -
ID: 299825199