Automatic Detection of Interpersonal Rapport

Current Projects: RAPT

Interpersonal closeness, or, rapport is an important element of human interaction, particularly in educational contexts, but current educational technologies are often unable to detect or respond to students in ways that draw on research on interpersonal social bonding in learning. In the "Rapport-Aligned Peer Tutor" (or, RAPT) project, we study rapport-building behaviors in peer tutoring in order to design a virtual peer tutor that can build rapport with students.

In this component of the work, we use the ground truth of a third-party rating of the rapport in each 30-second "thin-slice", with acceptable inter-rater reliability, and train a machine learning model to identify the temporal association rules for the verbal (tutoring and social) and nonverbal behaviors associated with each rapport level. We then use those rules as the input to a stacked ensemble classification model to predict the rapport level, given the presence of those behavioral sequences.

We published a paper in the 2017 Educational Data Mining conference describing this process, our prediction performance, and the nature of the temporal association rules we found for a variety of behavior types.

We also published a paper in the 2017 ACM Multimedia conference describing a neural-network based "temporally-selective attention model" for predicting rapport levels based on acoustic and visual features.