You are here

A Comparison of Emotion Annotation Approaches for Text

Authors: 

Ian Wood, John McCrae, Vladimir Andryushechkin , Paul Buitelaar

Publication Type: 
Refereed Original Article
Abstract: 
While the recognition of positive/negative sentiment in text is an established task with many standard data sets and well developed methodologies, the recognition of a more nuanced affect has received less attention: there are few publicly available annotated resources and there are a number of competing emotion representation schemes with as yet no clear approach to choose between them. To address this lack, we present a series of emotion annotation studies on tweets, providing methods for comparisons between annotation methods (relative vs. absolute) and between different representation schemes. We find improved annotator agreement with a relative annotation scheme (comparisons) on a dimensional emotion model over a categorical annotation scheme on Ekman’s six basic emotions; however, when we compare inter-annotator agreement for comparisons with agreement for a rating scale annotation scheme (both with the same dimensional emotion model), we find improved inter-annotator agreement with rating scales, challenging a common belief that relative judgements are more reliable. To support these studies and as a contribution in itself, we further present a publicly available collection of 2019 tweets annotated with scores on each of four emotion dimensions: valence, arousal, dominance and surprise, following the emotion representation model identified by Fontaine et al. in 2007.
Digital Object Identifer (DOI): 
10.3390/info9050117
ISSN: 
2078-2489
Publication Status: 
Published
Publication Date: 
11/05/2018
Journal: 
Information
Volume: 
9
Issue: 
5
Pages: 
117
Research Group: 
Institution: 
National University of Ireland, Galway (NUIG)
Open access repository: 
Yes