| 1 |
Identifying Causal Relationships Between Behavior and Local Brain Activity During Natural Conversation
|
|
|
|
In: Interspeech 2020 ; https://hal.archives-ouvertes.fr/hal-03016922 ; Interspeech 2020, Oct 2020, Shanghai, China. pp.101-105, ⟨10.21437/Interspeech.2020-2074⟩ (2020)
|
|
Abstract:
International audience ; Characterizing precisely neurophysiological activity involved in natural conversations remains a major challenge. We explore in this paper the relationship between multimodal conversational behavior and brain activity during natural conversations. This is challenging due to Functional Magnetic Resonance Imaging (fMRI) time resolution and to the diversity of the recorded multimodal signals.We use a unique corpus including localized brain activity and behavior recorded during a fMRI experiment when several participants had natural conversations alternatively with a human and a conversational robot. The corpus includes fMRI responses as well as conversational signals that consist of synchronized raw audio and their transcripts, video and eye-tracking recordings.The proposed approach includes a first step to extract discrete neurophysiological time-series from functionally well defined brain areas, as well as behavioral time-series describing specific behaviors. Then, machine learning models are applied to predict neurophysiological time-series based on the extracted behavioral features.The results show promising prediction scores, and specific causal relationships are found between behaviors and the activity in functional brain areas for both conditions, i.e., human-human and human-robot conversations.
|
|
Keyword:
[SCCO.COMP]Cognitive science/Computer science; [SCCO.LING]Cognitive science/Linguistics; [SCCO.NEUR]Cognitive science/Neuroscience; [SCCO]Cognitive science; Functional MRI; human-human and human-machine interactions; machine learning; multimodal signals processing; natural conversation
|
|
URL: https://hal.archives-ouvertes.fr/hal-03016922
https://hal.archives-ouvertes.fr/hal-03016922/document
https://doi.org/10.21437/Interspeech.2020-2074
https://hal.archives-ouvertes.fr/hal-03016922/file/hmamouche_INTERSPEECH2020.pdf
|
|
BASE
|
|
Hide details
|
|
|
| 2 |
BrainPredict: a Tool for Predicting and Visualising Local Brain Activity
|
|
|
|
In: Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020) ; https://hal.archives-ouvertes.fr/hal-03016059 ; Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pp.11 - 16, 2020 ; https://www.aclweb.org/anthology/2020.lrec-1.84/ (2020)
|
|
BASE
|
|
Show details
|
|
|
|
|
