SCSMI2017 Helsinki has ended
The annual conference of the Society for Cognitive Studies of the Moving Image (SCSMI) welcomes you to the Aalto University, Helsinki, Finland, June 11th – 14th, 2017

SCSMI2017 Helsinki program is under construction and changes are to occur. Meanwhile you may complete your personal information with a photo and some tags, so the other attendees and speakers will get to know more of you and your interests, and vice versa.

Go to registration or check practical information about accommodation etc. at http://scsmi2017.aalto.fi/
Back To Schedule
Tuesday, June 13 • 16:00 - 16:30
SP Jussi Tarvainen, Jorma Laaksonen and Tapio Takala. Film mood and its quantitative determinants in different types of scenes

Sign up or log in to save this to your schedule, view media, leave feedback and see who's attending!

Limited Capacity seats available

In mainstream cinema, emotions – both those displayed on screen and those elicited in the viewer – are central to the film viewing experience. One way that films seek to elicit emotions in viewers is by using various stylistic devices to infuse the story they tell with an affective character or tone – in a word, a mood. But the specific storytelling strategies used to build mood in a given film scene will depend on the type of scene in question: for example, dialogue is likely to have a greater influence on the mood of a dialogue scene than of an action scene, and most exterior scenes will provide more opportunities for mood-building through elaborate camera movement than interior scenes will.

Understanding how various narrative and stylistic attributes contribute to mood in different types of film scenes would provide insight into the way films engage and affect viewers. It would also guide the development of computational methods to estimate film mood based on quantitative features that can be detected from the film’s video and audio tracks. For example, if it turned out that the valence of a dialogue scene is largely determined by the facial expressions of the characters in the scene and the contents of their dialogue, then valence-modeling efforts should focus on developing computational features to describe those aspects of the scene.

In this study, we investigated the quantitative determinants of film mood across different types of scenes. We first investigated, through a user study, whether dimensions of film mood – hedonic tone (valence), energetic arousal, and tense arousal – can be assessed directly by viewers using continuous scales. We also investigated how the mood of film scenes is influenced by four groups of narrative and stylistic attributes: the events the scene depicts, the speech it contains, its visual style, and its use of sounds.

We then created a corpus of 50 scenes from various kinds of mainstream films and classified the scenes into discrete scene types using four criteria: location (interior, exterior, and mixed-location scenes), time of day (daytime and nighttime scenes), and the prominence of dialogue (dialogue and non-dialogue scenes) and music (music and non-music scenes). We then conducted another user study in which we collected style and mood ratings for each of the 50 scenes. This allowed us to investigate whether the mood ratings differed between scene types, and how well the ratings correlated with perceptual stylistic attributes assessed by viewers and features detected computationally from the scenes. The set of computational features we tested included both so-called low-level features that describe various stylistic attributes of the scene (e.g. brightness, fastness) as well as high-level features that describe the emotional expression in the faces, dialogue, and music contained in the scene. To obtain ground-truth data for testing the computational features, we manually tracked the movement of each character in the scenes, transcribed all the spoken dialogue, and marked the segments that contained dialogue or music.

The results of the studies showed, firstly, that direct assessment of film mood is feasible: the ratings exhibited high levels of internal consistency across all three mood dimensions. The results also indicated that the influence of stylistic attributes on hedonic tone is greater in non-dialogue scenes than in dialogue scenes, whereas stylistic attributes influence energetic and tense arousal in both of these scene types. We also found the hedonic tone ratings of non-dialogue and music scenes to be distributed evenly across the entire range of values from negative to positive, while dialogue and non-music scenes had Gaussian rating distributions, suggesting that strongly negative or positive moods are more likely to be found in the absence of dialogue or the presence of music. Lastly, we discovered that across all scene types, the energetic arousal dimension was associated with two stylistic attributes, loudness and fastness, and their corresponding low-level features, while hedonic tone and tense arousal were associated with high-level features that describe the emotional expression in faces, dialogue, and music. This finding was corroborated with linear regression analysis: models constructed with high-level features performed better with hedonic tone and tense arousal, and models constructed with low-level features performed better with energetic arousal.

In all, the results show that accounting for the distinctions between scene types can provide insight into the underpinnings of film mood under different conditions. The prominence of dialogue and music appear to be particularly useful scene type classification categories in this regard. The results also indicate that state-of-the-art computational features that describe the emotional expression in faces, dialogue, and music can be used to estimate a scene’s mood in terms of hedonic tone and tense arousal across various scene types. We have made the scene assessment and annotation data as well as the computational features publicly available.

avatar for Jussi Tarvainen

Jussi Tarvainen

Doctoral student, Aalto University School of Science, Department of Computer Science
I'm a doctoral student at the Aalto University School of Science. In my research I study film mood from the perspective of cognitive science and computer science. I'm interested in how mood in film is created, how it is perceived by viewers, and whether it can be estimated computationally... Read More →

Tuesday June 13, 2017 16:00 - 16:30 EEST
A-306 Room, Töölö Campus, Aalto University (3rd floor) Runeberginkatu 14-16, Helsinki