2 |
Incorporating nonverbal features into multimodal models of human-to-human communication
|
|
|
|
In: Theses and Dissertations Available from ProQuest (2008)
|
|
Abstract:
Nonverbal communication (e.g., eye gaze and hand gesture) plays an important role in human conversations, including providing semantic content, expressing emotional status, and regulating conversation turns. However, most analyses of conversations largely focus on spoken content and not important nonverbal cues. In this thesis, we investigated the use of nonverbal cues for enhancing the analysis of conversations. Particularly, we considered three types of conversational structures, speech repairs, sentence units, and floor control shifts, which are important for understanding conversations. To support the research on these three structures, we focused on: (1) collecting multimodal data resources involving nonverbal cues and structural events in human conversations, (2) analyzing the collected data sets to enrich our knowledge about nonverbal cues and the structural events investigated, and (3) building statistical models for detecting structural events using verbal and nonverbal cues. Through collaborations with researchers in psychology and computer vision, we collected the KDI multimodal dialog corpus and the VACE multimodal meeting corpus. Using the KDI data, we analyzed gesture patterns that occur during speech repairs. Using the VACE data, we analyzed gesture patterns for signaling the presence of SUs and gaze and gesture patterns for signaling floor control shifts. Using the VACE data, we then investigated combining gestural features with lexical and prosodic features for detecting SUs and combining gestural and gaze features for detecting floor control shifts. In this thesis, the impact of nonverbal cues was investigated for analysis and detection of structural events. The development of data resources and tools for this type of research is an important contribution. Our data analyses have enriched the knowledge about the relationship between nonverbal communication and structural events. Our statistical modeling research has demonstrated the usefulness of nonverbal cues for conversation analysis. Research in this thesis suggests that nonverbal communication provides useful cues for the analysis and detection of structural events in human conversations. Our results support the view that human conversations are processes involving multimodal cues, and so are more effectively analyzed using information from both verbal and nonverbal channels.
|
|
Keyword:
Electrical engineering
|
|
URL: https://docs.lib.purdue.edu/dissertations/AAI3342357
|
|
BASE
|
|
Hide details
|
|
4 |
Structural event detection for rich transcription of speech
|
|
|
|
In: Theses and Dissertations Available from ProQuest (2004)
|
|
BASE
|
|
Show details
|
|
7 |
The Effect of Pruning and Compression on Graphical Representations of the Output of a Speech Recognizer
|
|
|
|
In: Electrical and Computer Engineering Faculty Research and Publications (2003)
|
|
BASE
|
|
Show details
|
|
8 |
Statistical parsing and language modeling based on constraint dependency grammar
|
|
|
|
In: Theses and Dissertations Available from ProQuest (2003)
|
|
BASE
|
|
Show details
|
|
9 |
Investigating Probabilistic Constraint Dependency Grammars in Language Modeling
|
|
|
|
In: Department of Electrical and Computer Engineering Technical Reports (2001)
|
|
BASE
|
|
Show details
|
|
10 |
Familiarity and Pronounceability of Nouns and Names
|
|
|
|
In: Electrical and Computer Engineering Faculty Research and Publications (1999)
|
|
BASE
|
|
Show details
|
|
11 |
Parsing and tagging sentences containing lexically ambiguous and unknown tokens
|
|
|
|
In: Theses and Dissertations Available from ProQuest (1999)
|
|
BASE
|
|
Show details
|
|
14 |
Prosodic disambiguation in automatic speech understanding of Thai
|
|
|
|
In: Theses and Dissertations Available from ProQuest (1995)
|
|
BASE
|
|
Show details
|
|
15 |
Improving hidden Markov models for speech recognition
|
|
|
|
In: Theses and Dissertations Available from ProQuest (1995)
|
|
BASE
|
|
Show details
|
|
18 |
PARSEC: A Constraint-Based Parser for Spoken Language Processing
|
|
|
|
In: Department of Electrical and Computer Engineering Technical Reports (1993)
|
|
BASE
|
|
Show details
|
|
|
|