| 2 |
Sequence-to-Sequence Predictive Model: From Prosody To Communicative Gestures
|
|
|
|
In: Proceeding of the 24th International Conference on Human-Computer Interaction ; https://hal.archives-ouvertes.fr/hal-03428910 ; Proceeding of the 24th International Conference on Human-Computer Interaction, 2021 (2021)
|
|
Abstract:
International audience ; Communicative gestures and speech acoustic are tightly linked. Our objective is to predict the timing of gestures according to the acoustic. That is, we want to predict when a certain gesture occurs. We develop a model based on a recurrent neural network with attention mechanism. The model is trained on a corpus of natural dyadic interaction where the speech acoustic and the gesture phases and types have been annotated. The input of the model is a sequence of speech acoustic and the output is a sequence of gesture classes. The classes we are using for the model output is based on a combination of gesture phases and gesture types. We use a sequence comparison technique to evaluate the model performance. We find that the model can predict better certain gesture classes than others. We also perform ablation studies which reveal that fundamental frequency is a relevant feature for gesture prediction task. In another sub-experiment, we find that including eyebrow movements as acting as beat gesture improves the performance. Besides, we also find that a model trained on the data of one given speaker also works for the other speaker of the same conversation. We also perform a subjective experiment to measure how respondents judge the naturalness, the time consistency, and the semantic consistency of the generated gesture timing of a virtual agent. Our respondents rate the output of our model favorably.
|
|
Keyword:
[INFO.INFO-HC]Computer Science [cs]/Human-Computer Interaction [cs.HC]; Communicative Gesture; Machine Learning; Prosody
|
|
URL: https://hal.archives-ouvertes.fr/hal-03428910/file/hcii_2021_shortened.pdf
https://hal.archives-ouvertes.fr/hal-03428910
https://hal.archives-ouvertes.fr/hal-03428910/document
|
|
BASE
|
|
Hide details
|
|
|
| 3 |
Sequence-to-Sequence Predictive models: from Prosody to Communicative Gestures
|
|
|
|
In: Workshop sur les Affects, Compagnons artificiels et Interactions ; https://hal.inria.fr/hal-02933487 ; Workshop sur les Affects, Compagnons artificiels et Interactions, CNRS, Université Toulouse Jean Jaurès, Université de Bordeaux, Jun 2020, Saint Pierre d'Oléron, France (2020)
|
|
BASE
|
|
Show details
|
|
|
| 4 |
The ISO Standard for Dialogue Act Annotation, Second Edition
|
|
|
|
In: Proceedings of the 12th Language Resources and Evaluation Conference ; https://hal.archives-ouvertes.fr/hal-03011688 ; Proceedings of the 12th Language Resources and Evaluation Conference, May 2020, Marseille, France (2020)
|
|
BASE
|
|
Show details
|
|
|
| 5 |
Multimodal Analysis of Cohesion in Multi-party Interactions
|
|
|
|
In: LREC ; https://hal.archives-ouvertes.fr/hal-02953469 ; LREC, 2020, Marseille, France (2020)
|
|
BASE
|
|
Show details
|
|
|
| 6 |
A multimodal corpus of simulated consultations between a patient and multiple healthcare professionals
|
|
|
|
In: ISSN: 1574-020X ; EISSN: 1574-0218 ; Language Resources and Evaluation ; https://hal.sorbonne-universite.fr/hal-03209106 ; Language Resources and Evaluation, Springer Verlag, 2020, ⟨10.1007/s10579-020-09526-0⟩ (2020)
|
|
BASE
|
|
Show details
|
|
|
| 7 |
How Confident are You? Exploring the Role of Fillers in the Automatic Prediction of a Speaker's Confidence
|
|
|
|
In: Workshop sur les Affects, Compagnons artificiels et Interactions ; https://hal.inria.fr/hal-02933476 ; Workshop sur les Affects, Compagnons artificiels et Interactions, CNRS, Université Toulouse Jean Jaurès, Université de Bordeaux, Jun 2020, Saint Pierre d'Oléron, France (2020)
|
|
BASE
|
|
Show details
|
|
|
| 8 |
HOW CONFIDENT ARE YOU? EXPLORING THE ROLE OF FILLERS IN THE AUTOMATIC PREDICTION OF A SPEAKER’S CONFIDENCE
|
|
|
|
In: ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) ; https://hal.archives-ouvertes.fr/hal-02942182 ; ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), May 2020, Barcelona, Spain. pp.8104-8108, ⟨10.1109/ICASSP40776.2020.9054374⟩ (2020)
|
|
BASE
|
|
Show details
|
|
|
| 9 |
Disfluencies and Teaching Strategies in Social Interactions Between a Pedagogical Agent and a Student: Background and Challenges
|
|
|
|
In: SEMDIAL 2018 (AixDial), The 22nd workshop on the Semantics and Pragmatics of Dialogue ; https://hal.telecom-paris.fr/hal-02292437 ; SEMDIAL 2018 (AixDial), The 22nd workshop on the Semantics and Pragmatics of Dialogue, Nov 2018, Aix-en-Provence, France. pp.188-191 (2018)
|
|
BASE
|
|
Show details
|
|
|
| 11 |
Automating the Production of Communicative Gestures in Embodied Characters
|
|
|
|
BASE
|
|
Show details
|
|
|
| 12 |
Proceedings of the 18th International Conference on Intelligent Virtual Agents, Sydney, NSW, Australia, November 05 - 08, 2018
|
|
|
|
BASE
|
|
Show details
|
|
|
| 13 |
A Methodology for the Automatic Extraction and Generation of Non-Verbal Signals Sequences Conveying Interpersonal Attitudes
|
|
|
|
In: ISSN: 1949-3045 ; IEEE Transactions on Affective Computing ; https://hal.archives-ouvertes.fr/hal-01793271 ; IEEE Transactions on Affective Computing, Institute of Electrical and Electronics Engineers, 2017, XX, pp.1 - 1. ⟨10.1109/TAFFC.2017.2753777⟩ (2017)
|
|
BASE
|
|
Show details
|
|
|
| 14 |
Laughter Animation Generation
|
|
|
|
In: Handbook of Human Motion ; https://hal.sorbonne-universite.fr/hal-01528400 ; Bertram Müller; Sebastian I. Wolf; Gert-Peter Brueggemann; Zhigang Deng; Andrew McIntosh; Freeman Miller; William Scott Selbie. Handbook of Human Motion, Springer, 2017, ⟨10.1007/978-3-319-30808-1_190-1⟩ (2017)
|
|
BASE
|
|
Show details
|
|
|
| 15 |
The NoXi database: multimodal recordings of mediated novice-expert interactions
|
|
|
|
BASE
|
|
Show details
|
|
|
| 20 |
Analysis of Verbal and Nonverbal Communication and Enactment.The Processing Issues : COST 2102 International Conference, Budapest, Hungary, September 7-10, 2010, Revised Selected Papers
|
|
|
|
UB Frankfurt Linguistik
|
|
Show details
|
|
|
|
|
