Evaluating Deep Taylor Decomposition for Reliability Assessment in the Wild
Research output: Contribution to journal › Conference article › Research › peer-review
Standard
Evaluating Deep Taylor Decomposition for Reliability Assessment in the Wild. / Brandl, Stephanie; Hershcovich, Daniel; Søgaard, Anders.
In: Proceedings of the International AAAI Conference on Web and Social Media, Vol. 16, 2022, p. 1368-1372.Research output: Contribution to journal › Conference article › Research › peer-review
Harvard
APA
Vancouver
Author
Bibtex
}
RIS
TY - GEN
T1 - Evaluating Deep Taylor Decomposition for Reliability Assessment in the Wild
AU - Brandl, Stephanie
AU - Hershcovich, Daniel
AU - Søgaard, Anders
PY - 2022
Y1 - 2022
N2 - We argue that we need to evaluate model interpretability methods 'in the wild', i.e., in situations where professionals make critical decisions, and models can potentially assist them. We present an in-the-wild evaluation of token attribution based on Deep Taylor Decomposition, with professional journalists performing reliability assessments. We find that using this method in conjunction with RoBERTa-Large, fine-tuned on the Gossip Corpus, led to faster and better human decision-making, as well as a more critical attitude toward news sources among the journalists. We present a comparison of human and model rationales, as well as a qualitative analysis of the journalists' experiences with machine-in-the-loop decision making.
AB - We argue that we need to evaluate model interpretability methods 'in the wild', i.e., in situations where professionals make critical decisions, and models can potentially assist them. We present an in-the-wild evaluation of token attribution based on Deep Taylor Decomposition, with professional journalists performing reliability assessments. We find that using this method in conjunction with RoBERTa-Large, fine-tuned on the Gossip Corpus, led to faster and better human decision-making, as well as a more critical attitude toward news sources among the journalists. We present a comparison of human and model rationales, as well as a qualitative analysis of the journalists' experiences with machine-in-the-loop decision making.
U2 - 10.1609/icwsm.v16i1.19389
DO - 10.1609/icwsm.v16i1.19389
M3 - Conference article
VL - 16
SP - 1368
EP - 1372
JO - Proceedings of the International AAAI Conference on Web and Social Media
JF - Proceedings of the International AAAI Conference on Web and Social Media
SN - 2162-3449
T2 - 16th International AAAI Conference on Web and Social Media
Y2 - 6 June 2022 through 9 June 2022
ER -
ID: 339852192