Identifying the Function of Hand Gestures from their Form in Political Speech

Publikation: Bidrag til bog/antologi/rapportBidrag til bog/antologiForskningfagfællebedømt

This paper addresses the automatic classification of the semiotic type of hand gestures produced by six English speaking politicians in different contexts from the annotations of the gesture form and shape. It builds upon and extends to more data the work proposed in [24]. Gestures contribute to the successful delivery of a message in face-to-face communication by reinforcing what is conveyed by speech or adding new information to it. Gestures are multi-functional and can have different meanings depending on the context. The identification of the semiotic type of gestures is the first step towards their automatic interpretation. Moreover, exploiting the relation between gesture form and function, described in the literature, on many types of data, contributes to the automatic generation of gestures e.g. in infocommunication systems. In the present work we annotated the semiotic types of hand gestures produced by Boris Johnson in two question/answer sessions at the British House of Common and added it to the annotations of the gestures of five other politicians in different contexts. We trained various classifiers to identify the semiotic type of the hand gestures. The F1 score obtained by the best performing algorithm on the classification of four semiotic types is 0.68. This result confirms on a larger data set the results obtained in the preceding pilot study indicating that it is possible to identify the semiotic types of hand gestures from their coarse-grained form features in nearly one third of the cases.
TitelAccentuated Innovations in Cognitive Info-Communication
RedaktørerRyszard Klempous, Jan Nikodem, Péter Zoltán Baranyi
ISBN (Trykt)978-3-031-10955-3
ISBN (Elektronisk)978-3-031-10956-0
StatusUdgivet - 2022
NavnTopics in Intelligent Engineering and Informatics


ID: 322283097