LIRIS-ACCEDE

Database content

LIRIS-ACCEDE is the Annotated Creative Commons Emotional DatabasE. It is composed of six collections:

• Discrete LIRIS-ACCEDE - Induced valence and arousal rankings for 9800 short video excerpts extracted from 160 movies. Estimated affective scores are also available.
• Continuous LIRIS-ACCEDE - Continuous induced valence and arousal self-assessments for 30 movies. Raw and post-processed GSR measurements are also available.
• MediaEval 2015 Affective Impact of Movies collection - Violence annotations and affective classes for the 9800 excerpts of the discrete LIRIS-ACCEDE part, plus for additional 1100 excerpts used to extend the test set for the MediaEval 2015 affective impact of movies task.
• MediaEval 2016 Emotional Impact of Movies collection - Test set for the MediaEval 2016 Emotional Impact of Movies task: 1200 additional videos excerpts for the Global Annotation subtask and 10 additional movies for the Continous Annotation subtask.
• MediaEval 2017 Emotional Impact of Movies collection - Valence/arousal and fear annotations for the development and test sets of the MediaEval 2017 Emotional Impact of Movies Task. Visual and audio features are also provided.
• MediaEval 2018 Emotional Impact of Movies collection - Valence/arousal and fear annotations for the development and test sets of the MediaEval 2018 Emotional Impact of Movies Task. Visual and audio features are also provided.

The database is composed only of movies and excerpts from movies shared under Creative Commons licenses. The Creative Commons licenses allow creators to use standardized way to give the public permission to share and use their creative work under certain conditions of their choice. Creative Commons licenses consist of four major condition modules: Attribution (BY), requiring attribution to the original author; Share Alike (SA), allowing derivative works under the same or a similar license; Non-Commercial (NC), preventing the work from being used for commercial purposes; and No Derivative Works (ND), allowing only the use of the original work without any modification. Moreover, movies shared under Creative Commons licenses are often little known films, which limits viewers’ prior memories. The movies selected to be part of the LIRIS-ACCEDE are shared under Creative Commons licenses with BY, SA or NC modules. Movies shared with ND module are not taken into account in this work since it is not allowed to modify these videos, and therefore it is not allowed to segment them. Thus, using videos shared under Creative Commons licenses allows us to share the database publicly.

Discrete LIRIS-ACCEDE collection

160 films and short films with different genres are used and segmented into 9800 video clips. The total time of all 160 films is 73 hours 41 minutes and 7 seconds, and a video clip is extracted on average every 27s. The 9800 segmented video clips last between 8 and 12 seconds and are representative enough to conduct experiments. Indeed, the length of extracted segments is large enough to get consistent excerpts allowing the viewer to feel emotions and is also small enough to make the viewer feel only one emotion per excerpt. The content of the movie is also considered to create homogeneous, consistent and meaningful excerpts not to disturb the viewers. A robust shot and fade in/out detection has been implemented using to make sure that each extracted video clip start and end with a shot or a fade. Furthermore, the order of excerpts within a film is kept, allowing to study the temporal transitions of emotions. Several movie genres are represented in this collection of movies such as horror, comedy, drama, action and so on. Languages are mainly English with a small set of Italian, Spanish, French and others subtitled in English.

In order to sort the database along the induced valence and arousal axis, pairs of video clips were presented to annotators on Crowdflower. CrowdFlower is a crowdsourcing service which has over 50 labor channel partners, among them Amazon Mechanical Turk and TrialPay CrowdFlower differs from these individual networks because they offer enterprise solutions and a higher degree of quality control, called “Gold Standard Data”, to ensure the accuracy on the tasks.

Related publications:

• Y. Baveye, E. Dellandrea, C. Chamaret, and L. Chen, “LIRIS-ACCEDE: A Video Database for Affective Content Analysis,” in IEEE Transactions on Affective Computing, 2015.
• Y. Baveye, J.-N. Bettinelli, E. Dellandrea, L. Chen, and C. Chamaret, “A Large Video Database for Computational Models of Induced Emotion,” in 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction (ACII), 2013, pp. 13–18.
• Y. Baveye, E. Dellandrea, C. Chamaret, and L. Chen, “From crowdsourced rankings to affective ratings,” in IEEE International Conference on Multimedia and Expo Workshops (ICMEW), Jul. 2014, pp. 1–6.
• Y. Baveye, C. Chamaret, E. Dellandrea, and L. Chen, “A protocol for cross-validating large crowdsourced data: The case of the LIRIS-ACCEDE affective video dataset,” in Proceedings of the 2014 International ACM Workshop on Crowdsourcing for Multimedia, ser. CrowdMM ’14, 2014, pp. 3–8.

Continuous LIRIS-ACCEDE collection

The aim of this new experiment is to collect continuous annotations on whole movies. To select the movies to be annotated, we simply looked at the movies included in the LIRIS-ACCEDE dataset since they all share the desirable property to be shared under Creative Commons licenses and can thus be freely used and distributed without copyright issues as long as the original creator is credited. The total length of the selected movies was the only constraint. It had to be smaller than eight hours to create an experiment of acceptable duration. The selection process ended with the choice of 30 movies so that their genre, content, language and duration are diverse enough to be representative of the original LIRIS-ACCEDE dataset.

The annotation process aimed at continuously collecting the self-assessments of arousal and valence that viewers feel while watching the movies. To collect continuous annotations, we have used a modified version of the GTrace program originally developed by Cowie et al.. Annotations were collected from ten French paid participants (seven female and three male) ranging in age from 18 to 27 years (mean=21.9 +- 2.5 SD). Participants had different educational backgrounds, from undergraduate students to recently graduated master students.

Related publications:

• Y. Baveye, E. Dellandrea, C. Chamaret, and L. Chen, “Deep Learning vs. Kernel Methods: Performance for Emotion Prediction in Videos,” in 2015 Humaine Association Conference on Affective Computing and Intelligent Interaction (ACII), 2015.
• T. Li, Y. Baveye, C. Chamaret, E. Dellandrea, and L. Chen, “Continuous Arousal Self-assessments Validation Using Real-time Physiological Responses,” in ASM (ACM MM workshop), 2015.

MediaEval 2015 Affective Impact of Movies collection

The Affective Impact of Movies Task is part of the MediaEval 2015 Benchmarking Initiative. The overall use case scenario of the task is to design a video search system that uses automatic tools to help users find videos that fit their particular mood, age or preferences. To address this, we present two subtasks:

• Induced affect detection: the emotional impact of a video or movie can be a strong indicator for search or recommendation;
• Violence detection: detecting violent content is an important aspect of filtering video content based on age.

This year a single data set is proposed: 10,900 short video clips extracted from 199 Creative Commons-licensed movies of various genres. The movies are split into a development set – intended for training and validation – and a test set as 100, respectively 99 movies, resulting in 6,144 respectively 4,756 extracted short video clips. For each of the 10,900 video clips, the ground truth consists of: a binary value to indicate the presence of violence, the class of the excerpt for felt arousal (calm-neutral-active), and the class for felt valence (negative-neutral-positive).

Related publication:

• M. Sjöberg, Y. Baveye, H. Wang, V. L. Quang, B. Ionescu, E. Dellandréa, M. Schedl, C.-H. Demarty, and L. Chen, “The mediaeval 2015 affective impact of movies task,” in MediaEval 2015 Workshop, 2015.

MediaEval 2016 Emotional Impact of Movies collection

The Emotional Impact of Movies Task is part of the MediaEval 2016 Benchmarking Initiative. The task requires participants to deploy multimedia features to automatically predict the emotional impact of movies. We are focusing on felt emotion, i.e., the actual emotion of the viewer when watching the video, rather than for example what the viewer believes that he or she is expected to feel. The emotion is considered in terms of valence and arousal. Valence is defined as a continuous scale from most negative to most positive emotions, while arousal is defined continuously from calmest to most active emotions. Two subtasks are considered:

• Global emotion prediction: given a short video clip (around 10 seconds), participants’ systems are expected to predict a score of induced valence (negative-positive) and induced arousal (calm-excited) for the whole clip;
• Continuous emotion prediction: as an emotion felt during a scene may be influenced by the emotions felt during the previous ones, the purpose here is to consider longer videos, and to predict the valence and arousal continuously along the video. Thus, a score of induced valence and arousal should be provided for each 1s-segment of the video.

The development set is composed of the Discrete LIRIS-ACCEDE part for the first subtask, and the Continuous LIRIS-ACCEDE part for the second subtask. In addition to the development set, a test set is also provided to assess participants’ methods performance. 49 new movies under Creative Commons licenses have been considered. With the same protocol as the one used for the development set, 1,200 additional short video clips have been extracted for the first subtask (between 8 and 12 seconds), and 10 long movies (from 25 minutes to 1 hour and 35 minutes) have been selected for the second subtask (for a total duration of 11.48 hours).

Related publication:

• E. Dellandrea, L. Chen, Y. Baveye, M. Sjoberg and C. Chamaret, "The MediaEval 2016 Emotional Impact of Movies Task", in Working Notes Proceedings of the MediaEval 2016 Workshop, Hilversum, The Netherlands, October 20-21, 2016.

MediaEval 2017 Emotional Impact of Movies collection

The Emotional Impact of Movies Task is part of the MediaEval 2017 Benchmarking Initiative. The task requires participants to deploy multimedia features to automatically predict the emotional impact of movies. We are focusing on felt emotion, i.e., the actual emotion of the viewer when watching the video, rather than for example what the viewer believes that he or she is expected to feel. The emotion is considered in terms of valence, arousal and fear. Two new scenarios are proposed as subtasks. In both cases, long movies are considered and the emotional impact has to be predicted for consecutive 10-second segments sliding over the whole movie with a shift of 5 seconds:

• Valence/Arousal prediction: participants’ systems are supposed to predict a score of expected valence and arousal for each consecutive 10-second segments. Valence is defined as a continuous scale from most negative to most positive emotions, while arousal is defined continuously from calmest to most active emotions;
• Fear prediction: the purpose here is to predict for each consecutive 10-second segments whether they are likely to induce fear or not. The targeted use case is the prediction of frightening scenes to help systems protecting children from potentially harmful video content. This subtask is complementary to the valence/arousal prediction task in the sense that the mapping of discrete emotions into the 2D valence/arousal space is often overlapped (for instance, fear, disgust and anger are overlapped since they are characterized with very negative valence and high arousal).

The continuous part of LIRIS-ACCEDE is used as the development test for both subtasks. The test set consists of a selection of 14 movies other than the selection of the 160 original movies. They are between 210 and 6,260 seconds long. The total length of the 14 selected movies is 7 hours, 57 minutes and 13 seconds. The ground truth consists, for each 10-second segment, of a valence value, an arousal value and a binary value to indicate if the segment is supposed to induce fear or not.

Related publication:

• E. Dellandrea, Martijn Huigsloot, L. Chen, Y. Baveye and M. Sjoberg, "The MediaEval 2017 Emotional Impact of Movies Task", in Working Notes Proceedings of the MediaEval 2017 Workshop, Dublin, Ireland, September 13-15, 2017.

MediaEval 2018 Emotional Impact of Movies collection

The Emotional Impact of Movies Task is part of the MediaEval 2018 Benchmarking Initiative. The task requires participants to deploy multimedia features to automatically predict the emotional impact of movies. We are focusing on felt emotion, i.e., the actual emotion of the viewer when watching the video, rather than for example what the viewer believes that he or she is expected to feel. The emotion is considered in terms of valence, arousal and fear. Two new scenarios are proposed as subtasks. In both cases, long movies are considered:

• Valence/Arousal prediction: participants’ systems have to predict a score of expected valence and arousal continuously (every second) along movies. Valence is defined as a continuous scale from most negative to most positive emotions, while arousal is defined continuously from calmest to most active emotions;
• Fear prediction: the purpose here is to predict beginning and ending times of sequences inducing fear in movies. The targeted use case is the prediction of frightening scenes to help systems protecting children from potentially harmful video content.

A total of 44 movies (total duration of 15 hours and 20 minutes) selected from the set of 160 movies of the LIRIS-ACCEDE dataset are provided as development set for both subtasks with the annotations according to fear, valence and arousal. A complementary set of 10 movies (11 hours and 29 minutes) is available for the first subtask with the valence and arousal annotations. The test set consists of 12 other movies selected from the set of 160 movies, for a total duration of 8 hours and 56 minutes. In addition to the video data, participants are also provided with general purpose audio and visual content features. The ground truth consists of valence and arousal values for each second of each movie (first subtask) and the beginning and ending times of each sequence inducing fear in each movie (second subtask).

Related publication:

• E. Dellandrea, M. Huigsloot, L. Chen, Y. Baveye, Z. Xiao and M. Sjoberg, "The MediaEval 2018 Emotional Impact of Movies Task", in Working Notes Proceedings of the MediaEval 2018 Workshop, Sophia Antipolis, France, October 29-31, 2018.

List of movies