Lisbon Emoji and Emoticon Database (LEED): Norms for emojiand emoticons in seven evaluative dimensions

David Rodrigues1,2 & Marília Prada1 & Rui Gaspar3 & Margarida V. Garrido1 &

Diniz Lopes1

Published online: 31 March 2017# Psychonomic Society, Inc. 2017

Abstract The use of emoticons and emoji is increasingly pop-ular across a variety of new platforms of online communica-tion. They have also become popular as stimulus materials inscientific research. However, the assumption that emoji/emoticon users’ interpretations always correspond to the de-velopers’/researchers’ intended meanings might be mislead-ing. This article presents subjective norms of emoji and emo-ticons provided by everyday users. The Lisbon Emoji andEmoticon Database (LEED) comprises 238 stimuli: 85 emoti-cons and 153 emoji (collected from iOS, Android, Facebook,and Emojipedia). The sample included 505 Portuguese partic-ipants recruited online. Each participant evaluated a randomsubset of 20 stimuli for seven dimensions: aesthetic appeal,familiarity, visual complexity, concreteness, valence, arousal,and meaningfulness. Participants were additionally asked toattribute a meaning to each stimulus. The norms obtained in-clude quantitative descriptive results (means, standard devia-tions, and confidence intervals) and a meaning analysis foreach stimulus. We also examined the correlations betweenthe dimensions and tested for differences between emoticons

and emoji, as well as between the two major operating sys-tems—Android and iOS. The LEED constitutes a readilyavailable normative database (available at www.osf.io/nua4x)with potential applications to different research domains.

Keywords LEED . Emoticons . Emoji . Aesthetic appeal .

Familiarity . Visual complexity . Concreteness . Valence .

Arousal . Meaningfulness . Meaning analysis . Normativeratings . Android . iOS . Facebook . ICTs

Human communication involves the transmission of abstractand concrete information using both verbal and nonverbal sym-bols (for a review, see Richmond & McCroskey, 2009). In thelast few decades, and particularly as of the beginning of the 21stcentury, innovations in technology have dramatically changedthe ways that people communicate with each other. The increas-ing worldwide Internet usage and smartphone ownership, in-cluding in emerging economies (PEW Research Center, 2016),has introduced different forms of written communication medi-ated by information and communication technologies (ICTs).These include instant messaging and e-mail applications basedon ICT-device operating systems (Android, iOS) or messagingservices (e.g., Gmail, Whatsapp), VoIP system providers (e.g.,Skype), social networking sites (e.g., Facebook), and social me-dia platforms (e.g., Twitter).

Some authors have suggested that these forms of commu-nication filter out social, affective, and nonverbal/visual cuesand can originate less effective communication outcomes(e.g., Walther, 1996; Walther & D’Addario, 2001). However,other studies have shown that the absence of such cues doesnot necessarily render communications less effective. Instead,this absence may promote the implementation of uncertaintyreduction strategies to compensate for the absence (Antheunis,Valkenburg, & Peter, 2007, 2010). In particular, the use of

Electronic supplementary material The online version of this article(doi:10.3758/s13428-017-0878-6) contains supplementary material,which is available to authorized users.

* David Rodriguesdflrs@iscte.pt

1 Department of Social and Organizational Psychology, InstitutoUniversitário de Lisboa (ISCTE-IUL), CIS-IUL, Av. das ForçasArmadas, Office AA121, 1649-026 Lisbon, Portugal

2 Goldsmiths, University of London, London, UK3 William James Center for Research, ISPA - Instituto Universitáriov,

Lisbon, Portugal

Behav Res (2018) 50:392–405DOI 10.3758/s13428-017-0878-6

written paralanguage cues in written communication, has beenidentified as a strategy to overcome the absence of certaincues, because they convey meaning (e.g., Lea & Spears,1992). These cues include typographical marks (i.e., lettersand numbers) and ideograms (e.g., graphic symbols), identi-fied as Btypographic or text-based emoticons^ and Bgraphicemoticons,^ respectively (e.g., Huang, Yen, & Zhang, 2008;Wang, Zhao, Qiu, & Zhu, 2014). In the late 1990s, the latteremerged as an independent strand of meaning and emotionalexpression through ideograms and pictographs that could beused across ICT platforms. These came to be known as emoji,created with the goal of facilitating mobile communication(Negishi, 2014; Nelson, Tossell, & Kortum, 2015).

In addition to their massive use in daily written commu-nications, both emoticons and emoji are being used increas-ingly in applied domains, such as marketing and health, aswell as stimulus materials in scientific research (e.g.,Davidov, Tsur, & Rappoport, 2010; Hogenboom et al.,2013; Skiba, 2016; Thelwall, Buckley, & Paltoglou, 2012;Thelwall, Buckley, Paltoglou, Cai, & Kappas, 2010;Vashisht & Thakur, 2014; Wang & Castanon 2015).However, their selection, coding, and analysis may be some-what biased if we assume a direct correspondence betweenthe users’ interpretations of emoji/emoticons and theirintended meanings (e.g., a sad face emoji is negative andwill be perceived as such).

In this study we report evaluations of emoticons and emojiprovided by ICT users. Specifically, we present the LisbonEmoji and Emoticon Database (LEED), and provide the firstset of normative evaluations for 238 stimuli, comprising 85emoticons and 153 emoji, based on seven evaluative dimen-sions: aesthetic appeal, familiarity, visual complexity, concrete-ness, valence, arousal, and meaningfulness. In addition, weexamined the meaning attributed to each stimulus. It is ourcontention that the LEED contributes to the literature by pro-posing subjective norms for emoji and emoticons andguaranteeing the quality of the codebooks used in both re-search and practice in a multitude of areas.

Emoticons and emoji in ICT-mediatedcommunication

Emoticons and emoji have been considered a new medium toshare daily narratives, emotions, and attitudes with othersthrough ICTs (for a review, see Gülşen, 2016). Emoticons(from emotion + icon) are symbols created by using punctua-tion, numbers, or letters, with the intention of transmitting feel-ings, emotional states, or information in the absence of words,or complementing a written message (Dresner & Herring,2010; Krohn, 2004; Thompson& Filik, 2016). The first knownemoticons :( and :)were proposed in 1982 and are attributed toScott E. Fahlman, a professor at Carnegie Mellon’s School of

Computer Science, who created them in an attempt to differ-entiate serious posts from joke remarks on a bulletin board.1

Since then, emoticons have hugely increased in number, andthe current list of emoticons is extensive, running from simplesymbols to highly complex ones (e.g., www.netlingo.com/smileys). Emoticons include representations of facialexpressions, typically sideways [Western style; e.g., ;)], aswell as representations of abstract concepts and emotions/feelings (e.g., <3). Other emoticons are represented in aright-way-up position [Eastern style; e.g., (*^.^*)].

Emoji (from the Japanese e [picture] +moji [character]) aregraphic symbols with predefined names/IDs and code(Unicode), which include not only representations of facialexpressions (e.g., ), abstract concepts (e.g., ), andemotions/feelings (e.g., ), but also animals (e.g., ), plants(e.g., ) activities (e.g., ), gestures/body parts (e.g., ), andobjects (e.g., ). Emoji are presumed to have been first pro-posed by Shegetaka Kurita during the late 1990s, who createdthem while working at a mobile phone operator in Japan tofacilitate mobile communication (Negishi, 2014). Currently,more than 2,000 emoji are supported by different platforms,and they are constantly evolving and becoming more diverse(http://emojipedia.org). For instance, new Unicode releases(e.g., Unicode 11.0, released in 2016) include emoji thatrepresent different social groups—varying, for example, inethnicity (e.g., ) and age (e.g., ).

There are major differences between emoji and emoticons(Ganster, Eimler and Kramer, 2012). As compared to emoti-cons, emoji are colored, are not rotated by 90°, and in thoserepresenting facial expressions, the face is often delimited by acircle and may include multiple facial cues.

However, both emoticons and emoji are increasingly popu-lar in our everyday life. They are a constant presence in theways we communicate in the virtual world (e.g., social media,e-mail, and text messages; Gülşen, 2016). Emoji are also beingincluded in everyday products (e.g., toys, home decorationitems, or even clothes). Moreover, emoji have been integratedin the ways artists communicate with their audience (e.g., KatyPerry’s BRoar^music video) and the ways brands connect withconsumers (for a review, see Wohl 2016). For instance, brandshave included emoji in advertising campaigns (e.g.,McDonalds used people with emoji as their heads; Beltrone,2015) and developed new sets of brand-related emoji (e.g.,Dove launched a set of curly-haired emoji; Neff, 2015). Inanother example of emoji popularity, the Oxford dictionariesconsidered the emoji Bface with tears of joy^ to be Btheword of the year 2015.^On Twitter alone, this emoji registered6.6 billion uses that year (@TwitterData).

Scientific research about emoticons and emoji is also in-creasing. Some studies have examined naturalistic data, such

1 For a first-person account of emoticon history, see www.cs.cmu.edu/~sef/sefSmiley.htm.

Behav Res (2018) 50:392–405 393

as public messages posted on social media platforms (e.g.,Twitter, Google forums, or Facebook) to understand and char-acterize emoticon/emoji usage. For example, Novak,Smailović, Sluban, and Mozetič (2015) proposed the emojisentiment ranking, an index of positivity based on the frequen-cy of each emoji used in negative, neutral, and positive tweets.Also, Ljubešić and Fišer (2016) used tweets as their dataset toinvestigate how popular emoji are on Twitter, which countriesexhibit greater emoji usage, and the popularity of specificemoji. Similarly, Tossell and colleagues (2012) conducted alongitudinal study monitoring the use of emoticons in textmessages. This type of descriptive analysis can also be con-ducted in specific domains. For example, Vidal, Ares, andJaeger (2016) examined tweets about eating situations andhow people used emoticons/emoji to spontaneously expressfood-related emotional experiences. Other studies used similarnaturalistic data to monitor a given event (e.g., public healthinformation; Paul & Dredze, 2011) or to examine event-centered reactions, opinions, feelings, evaluations, or emotions(e.g., elections; Burnap, Gibson, Sloan, Southern, &Williams,2016). Even though these studies have typically relied on emo-tional word lexicons, more recently researchers have drawnattention to the need to extend these lexicons to include emo-ticons and emoji (B. Liu, 2012; Pang & Lee, 2008).

Research focusing on emoticon/emoji usage and functionshas suggested that these stimuli serve two key functions: toportray emotional or social intent and to reduce potential dis-course ambiguity (for a review, see Kaye, Wall, & Malone,2016). Skovholt, Grønning, and Kankaanranta (2014) showedthat such stimuli also function as contextualization cues (e.g.,markers of positive attitudes that facilitate message interpre-tation) and as organizers of social relationships in written in-teraction (e.g., reducing perceived interpersonal distance bydecreasing impersonality/formality). As examples of thesefunctions, Lo (2008) showed that adding emoticons to onlinemessages improved receivers’ understanding of the intensityand valence of the emotions (sad vs. happy) and attitudes (likevs. dislike) expressed by the sender. Likewise, Ganster andcolleagues (2012) showed that using a smiling (vs. afrowning) emoji/emoticon influences how a message is eval-uated (i.e., more positive and humorous), how the sender isperceived (i.e., more extroverted), and how the receiver feels(i.e., a more positive mood). Derks, Bos, and von Grumbkow(2008) further showed that emoticons strengthen the intensityof a message (e.g., a positive message with a smile emoticon israted more positively than the same positive message withoutthe emoticon). However, in the case of incongruence betweenthe valences of the message and the emoticon (e.g., a positivemessage accompanied by a frown emoticon), a message’s in-terpretation relies more on the text content.

Another line of research has adopted experimental meth-odologies to examine how the presentation of emoticons/emoji influences different phenomena. For example, Wang

and colleagues (2014) focused on the effects of adding posi-tive and negative emoji to messages regarding workplace per-formance on acceptance of negative feedback. Likewise, Tungand Deng (2007) tested how the presentation of emoji in an e-learning environment affected children’s motivation.Furthermore, Siegel and colleagues (2015) investigatedwhether including emoji on food packages influenced chil-dren’s meal choices. Emoji and/or emoticons have also beenused as the experimental materials in studies focusing on af-fective processing (e.g., Han, Yoo, Kim, McMahon, &Renshaw, 2014; Kerkhof et al., 2009; Yuasa, Saito, &Mukawa, 2011). For example, positive and negative emojihave been used as primes to induce valence, influencing re-sponses (event-related potentials) to valenced target words(e.g., Comesaña et al., 2013). Research has shown that noveltarget words primed with positive emoji are more likely to beerroneously categorized as familiar (e.g., Garcia-Marques,Mackie, Claypool, & Garcia-Marques, 2004). Finally, emoji/emoticons have been used for research method develop-ment—for example, as anchors in rating scales assessing cur-rent emotional states (e.g., Moore, Steiner, & Conlan, 2013),emotional associations with specific stimuli (e.g., food names;Jaeger, Vidal, Kam, & Ares, 2017), well-being (Fane,MacDougall, Jovanovic, Redmond, & Gibbs, 2016), and pain(e.g., Chambers & Craig, 1998).

Methodologies and tools for emoticons/emoji analysis

The selection, coding, and analysis of emoticons and emoji asdirect indicators of the emotional meanings conveyed by mes-sages can follow either human-based (e.g., Park, Baek, & Cha,2014; Vidal et al., 2016) or computer-based (Davidov et al.,2010; Hogenboom et al., 2013; Vashisht & Thakur, 2014; H.Wang & Castanon, 2015) procedures. A computer-based pro-cedure relies on machine-learning algorithms and semanticlexicons that is thought to provide a more objective analysisof emoticon/emoji usage. Both human-based and computer-based procedures may be prone to bias because they rely ex-clusively on the evaluations of, and the meanings attributed byresearchers/analysts, without taking into consideration theways they are perceived by the users. One area in which thishas been particularly worrisome is the field of computer-basedsentiment analysis (Thelwall et al., 2012; Thelwall et al.,2010), which allows for detecting and analyzing sentiment/affective reactions on the basis of semantic analysis of writtentext. Such analyses rely on codebooks developed by re-searchers from of the commonly accepted designations/feelings portrayed by emoticons and emoji (e.g., emoticon-smoothed language models; Liu, Li, & Guo, 2012;SentiStrenght coding manual for sentiment in texts availableat http://sentistrength.wlv.ac.uk; e.g., Thelwall et al., 2012;Thelwall et al., 2010).

394 Behav Res (2018) 50:392–405

The emoji sentiment ranking (Novak et al., 2015) consti-tutes an attempt to overcome some of these limitations.However, this index focuses exclusively on the valence dimen-sion and does not take into account other relevant information,such as the level of arousal elicited by a given emoji or themeaning attributed to it. Therefore, standardized proceduresfor the classification of emoticons/emoji are still missing.

In our view, this state of affairs may have two potentialproblems. First, the stimulus selection, coding, and analysismay be prone to biases due to researchers’ own evaluations ofthe stimuli (e.g., analyses based on ad-hoc emotionality catego-rization made by two coders; Park et al., 2014). Second, theremay be a biased assumption that emoticon/emoji users’ inter-pretations necessarily correspond to the meanings intended bydevelopers/researchers. Because emoji/emoticons are not usu-ally labeled when presented (with the exception of theFacebook emoji set), they are open to interpretation. Indeed,users can select an emoji on the basis of superficial visual fea-tures, which can lead to misinterpretations of its meaning andintent. For example, one may wish to express sadness byselecting a tearful emoji, and mistakenly choose Bface withtears of joy^ instead of Bface with tears of sadness^.Additionally, the same emoticon/emoji can be used to representa variety of meanings. For instance, a smiley face may be usedto express happiness, but it may also be used to express agree-ment with or liking of something/someone, one’s own physicalor mental well-being state, empathy, comprehension, or othermeanings. Moreover, the same emoticon/emoji can beinterpreted differently according to the communication context.For example, emoticons such as :p and ;) are typically de-scribed as positive, but they can also be used as markers ofirony (Carvalho, Sarmento, Silva, & de Oliveira, 2009) or sar-casm (Thompson & Filik, 2016). Finally, emoji with the sameintended meaning may have distinct visual representationsacross operating systems, potentially leading to different inter-pretations and evaluations (Miller et al., 2016). To sum up, aswith other types of visual stimuli, emoji/emoticons are prone tosubjectivity in their evaluation and interpretation, which sup-ports the need to develop a normative database.

Normative data are abundant in the literature (for reviews,see Prada, Rodrigues, Silva, & Garrido, 2015; Proctor & Vu,1999). These validated databases typically include stimulisuch as words (e.g., Bradley & Lang, 1999a), sounds (e.g.,Bradley & Lang, 1999b), or images depicting a broad range ofcontents (e.g., Dan-Glauser & Scherer, 2011; Lang, Bradley,& Cuthbert, 2008). Regarding the latter type of stimuli, somedatabases include, for example, visual materials such as sim-ple line drawings (e.g., Bonin, Peereman, Malardier, Méot, &Chalard, 2003; Snodgrass & Vanderwart, 1980) or symbols(e.g., McDougall, Curry, & Bruijn, 1999; Prada et al., 2015).Other databases are theme-focused and include specific con-tents, such as food (e.g., Blechert, Meule, Busch, & Ohla,2014; Charbonnier, van Meer, van der Laan, Viergever, &

Smeets, 2016) or human faces (e.g., Ebner, Riediger, &Lindenberger, 2010; Garrido et al., 2016; Mendonça,Garrido, & Semin, 2016).

The absence of published normative data on visual stimulisuch as emoticons and emoji has two important consequences.First, it implies that researchers should make the additionaleffort of pretesting materials to meet a study’s demands. Forexample, prior to their affective-priming study, Comesaña andcolleagues (2013) had to conduct two extensive pretests inwhich 180 participants evaluated the valence, arousal, andmeaning associated with each emoji. Second, the comparisonof results between studies can be challenging because thestimuli are often categorized ad hoc. For example, in theirstudy on tweets about food, Vidal and colleagues (2016) hadtwo coders categorizing the emoji and emoticons as negative,neutral, or positive by considering their intended meaning oravailable description. Park and colleagues (2014) also had twocoders categorizing emoticons on three levels, but they consid-ered a different dimension (i.e., emotionality: sad, neutral, andhappy) and distinct criteria (emotions conveyed by shape of theeyes and by the shape of the mouth).

In the present article, we present normative ratings of a setof emoticons and emoji from the two most used operatingsystems—Android and iOS. We also included Breaction^emoji from the most used social networking platform—Facebook. Each stimulus was evaluated with regards to itsaesthetic appeal, familiarity, visual complexity, semantic clar-ity, the valence and arousal of the meaning conveyed, andmeaningfulness. Additionally, we assessed the subjectivemeanings attributed by participants to each stimulus. We se-lected this set of seven evaluative dimensions on the basis ofprevious norms with other types of visual stimuli. Specifically,we followed the methodology adopted in a recent validationstudy (for a detailed review of the dimensions of interest, seePrada et al., 2015), with the exception of adding the dimensionof clarity, which has emerged as being relevant for the evalu-ation of facial expressions (for a review, see Garrido et al.,2016).

Method

Participants

A sample of 505 Portuguese individuals (71.7% women;Mage = 31.10, SD = 12.70) volunteered to participate in aWeb survey. These individuals were recruited online throughFacebook (university institutional page and online studies ad-vertisement pages) and mailing services (students mailinglists). All participants were native Portuguese speakers orhad lived in Portugal for the last 5 years. The sample com-prised mostly university students (46.7%) and active workers(43.3%), with at least a bachelor degree (46.0%). Most

Behav Res (2018) 50:392–405 395

participants indicated that Android/Google (67.5%) or iOS(26.3%) was their operating system.

Stimulus set

The LEED includes 238 stimuli2: 85 emoticons and 153 emoji(77 from iOS, 63 from Android, 9 from Facebook, and 4 fromEmojipedia), mostly representing facial expressions of emo-tions (e.g., Bhappy face^) and/or symbolic meanings (e.g.,Bsilence^).3

The emoticon set was developed on the basis of the list ofemoticons presented in the BTwitter emotion codinginstructions^4 for the SentiStrenghth tool (Thelwall et al.,2010; adapted from Wiebe, Wilson, & Cardie, 2005), usedfor sentiment detection in short texts. This list included 63Western-style emoticons (e.g., Emot07; see Fig. 1) and 23Eastern-style emoticons (e.g., Emot56a; see Fig. 1). One sym-bol was removed due to its unavailability in mobile phone textpackages (:Þ).

Because a given emoticon can sometimes vary in its pre-sentation, variations of the same stimulus were included. Forexample, Emot01 (Blaughing, big grin^) has three variationsidentified in the database, from Emot01a to Emot01c. Eachemoticon was generated in black, 28-point Arial font on awhite background and was saved as a single image file(72 × 68 pixels, 72 dots per inch, RGB, PNG format).

According to information available from the UnicodeFoundation (http://unicode.org/emoji/charts/full-emoji-list.html), we selected emoji with intended meanings similar tothe emoticons. Figure 1 depicts examples of the emoticonsand emoji for Blaughing^ and Bcrying.^ As in the case ofemoticons, variations of the same emoji were included. Theset of 153 emoji was extracted from the Emojipedia database(http://emojipedia.org/) and included stimuli from the twomost used and available operating systems at the time thestudy was performed: Apple iOS 9.3 (used in iPhone, iPad,iMac, Apple Watch, and Apple TV) and Google Android 6.0.1 (used in Android devices, the Gmail Web interface, GoogleHangouts, and the Google Chrome Internet browser).

Emoji were matched across operating systems according totheir Unicode references. Of the 153 emoji, 63 stimuli wererepresented in both operating systems (all 63 Android emojihad a corresponding iOS emoji), 14 were only represented inthe iOS operating system (e.g., EmjAp51), and 8 were repre-sented in both operating systems and in the Facebook reaction

set (see Fig. 1).5 The latter subset included nine emoji: thelike/dislike buttons (EmjFb76 and EmjFb77, respectively),the recently added BFacebook reactions^ (five faces express-ing emotions, EmjFb07–EmjFb67, and one heart symbol,EmjFb71), and the new Blike^ button (EmjFb78). Finally, fourEmojipedia images (EmjPe86–EmjPe89) were also includedin the final set. These Unicode 9.0 emoji were not available inthe Android or iOS operating systems at the time of the study(e.g., EmjPe89), but were included to represent potentiallyofficial future emoji not currently available. Each emoji wassaved as a single image file (72 × 72 pixels, 72 dots per inch,RGB, PNG format).

The vast majority of the emoji set represents facial expres-sions (88.89%), with the exceptions of popular symbols(3.27%; e.g., heart, EmjAn71; heartbreak, EmjAn72) andhand gestures (7.84%; e.g., hand palm, EmjAp75).

Procedure and measures

The study was conducted using the Qualtrics software.Participants were invited to collaborate on a web survey aboutthe perception and evaluation of emoticons and emoji. Afterclicking on the hyperlink, participants were directed to a se-cure webpage and were informed about the goals of the studyand its expected duration (approximately 20 min). The initialinstructions provided the definition of all emoji and emoti-cons, and examples of each type of stimulus were presented(emoticons and emoji ). To avoid overlap,these examples were different from the stimuli used in theevaluation task. Participants were also informed that all thedata collected would be treated anonymously and that theycould abandon the study at any point by closing the browser,without their responses being considered for analysis.

After providing their informed consent to collaborate inthe study (by checking the BI agree^ option), participantswere asked to provide information regarding their age, sex,educational level, current occupation, and their operatingsystem. Following this, they were given specific instruc-tions to evaluate each stimulus in seven evaluative dimen-sions, namely: aesthetic appeal, familiarity (subjective fre-quency), visual complexity, clarity, valence, arousal, andmeaningfulness (all dimensions rated using 7-point Likert-type scales; the detailed instructions for each scale are pre-sented in Table 1; see also Garrido et al., 2016; Prada et al.,2015). These dimensions were randomly presented per trial

2 The full set of stimuli is available as online supplemental material, and atwww.osf.io/nua4x. This includes the corresponding Unicode references(http://unicode.org/emoji/charts/full-emoji-list.html) and intended meaningsfor each emoticon/emoji proposed by the Unicode Foundation.3 For identifying the stimuli in our database, we used the prefixes Emot =Emoticon, Emj = Emoji, Ap = Apple iOS, An = Google Android, Fb =Facebook, Pe = Emojipedia.4 Avai lable a t h t tp : / / sent i s t rength .wlv.ac .uk/documenta t ion/TwitterVersionOfSentimentCodeBook.doc

5 This excludes the new Blike^ emoji, which had no correspondence, giventhat the old Blike^ button was the one used as a correspondent to similar emojiin iOS and Android. Moreover, a nonexistent emoji in Facebook, the Bdislike^emoji (representing the Blike^ emoji in an inverted position), was also includedin the stimulus set, given that when the study materials were created the newswas stating that Facebook would include this option in their platform, whichlater proved not to be the case.

396 Behav Res (2018) 50:392–405

in the evaluation task. Finally, participants were requestedto write the first meaning or emotion that came to theirmind for each stimulus in an open-ended response format,or alternatively select the option BI don't know^ if theywere not able to provide a specific meaning or emotion.The instructions also emphasized that responses wouldhave to be fast and spontaneous and that there were noright or wrong answers.

Participants then proceeded to the main task. To preventfatigue and demotivation, each participant only saw a sub-set of 20 randomly selected stimuli from the available poolof 238 stimuli. Each stimulus was presented on a singlepage of the Web survey. We used a forced response option,such that participants were required to answer each ques-tion to progress in the survey. The number of participantsevaluating each stimulus varied from 40 to 49. The stimuliwere always presented at the top left corner of the page,with all evaluative dimensions presented below. Uponcompleting the task, participants were thanked anddebriefed.

Results

The norms for the full set of stimuli are provided as supple-mentary material. (see also www.osf.io/nua4x) In thefollowing sections we present (a) the preliminary analysis re-garding outlier detection, (b) the analysis of the differences bygender and operating system, (c) the subjective rating normsfor each dimension, (d) the correlations between evaluativedimensions, and (e) the analysis of attributed meaning/emotion.

Preliminary analysis

Because only completed surveys were included in the analy-sis, there were no missing data. Outliers were determined interms of the criterion of 2.5 standard deviations above or be-low the mean evaluation of each stimulus on a given dimen-sion. This analysis yielded a small percentage (1.32%) of out-lier ratings. Moreover, none of the participants responded

Table 1 Instructions and scale anchors for each dimension

Dimension Instructions Scale

1. Aesthetic appeal In your opinion, considering the visual characteristics of the symbol, and not theobject or concept it may depict, how visually appealing is the stimulus?

1 = Visually unpleasant/unappealing,7 = Visually very pleasant/appealing

2. Familiarity How frequently do you encounter or see this stimulus in your daily routine? Morefrequently encountered stimuli are more familiar.

1 =Not familiar,7 = Very familiar

3. Visual complexity Considering the complexity of the visual characteristics of the stimulus, and notthose of the concept that can be related to it, how much visual detail andcomplexity does this stimulus contain? The more details the stimulus contains,the more complex it is.

1 = Very simple,7 = Very complex

4. Clarity How clear or ambiguous is this stimulus? Stimuli that, in your opinion, clearlyconvey an emotion/meaning should be considered clear. Otherwise, theyshould be considered more ambiguous.

1 = Totally ambiguous,7 = Totally clear

5. Valence To what extent do you consider this stimulus refers to something positive/pleasantor negative/unpleasant.

1 = Very negative,7 = Very positive

6. Arousal To what extent do you consider this stimulus refers to somethingarousing/exciting or passive/calm?

1 = Very passive/calm,7 = Very arousing/exciting

7. Meaningfulness Please indicate to what extent this stimulus conveys a meaning/emotion. 1 =Conveys no meaning/emotion at all,7 =Conveys a lot of meaning/emotion

Fig. 1 Sample emoticons and emoji across operating systems for Blaughing^ and Bcrying^ (the stimulus codes are included)

Behav Res (2018) 50:392–405 397

systematically in the same way (i.e., using the same value ofthe scale). Therefore, no participants were excluded.

Emoticons and emoji evaluations

When we compared the evaluations of emoticons and emojion each dimension for the total sample (see Table 2), theoverall results showed that emoji (vs. emoticons) were ratedas aesthetically more appealing, t(498) = –24.82, p < .001, d =1.11; more familiar, t(498) = –23.73, p < .001, d = 1.06; clear-er, t(497) = –31.45, p < .001, d = 1.41; more positive,t(498) = –2.50, p = .013, d = 0.11; more arousing, t(498) = –21.51, p < .001, d = 0.96; and more meaningful, t(498) = –31.00, p < .001, d = 1.39.

Gender differences The general results of the comparisonbetween emoticons and emoji were also observed in the sub-samples of both women and men. However, men providedequivalent valence ratings for the emoticons and emoji. Wealso tested for gender differences in the evaluations of emoti-cons and emoji for each dimension. As shown in Table 2, nogender differences emerged in the ratings of emoticons. Whenwe replicated this analysis for emoji, the results showed thatwomen evaluated emoji as being more familiar, clear, andmeaningful than did men, all ps ≤ .015. This pattern of results

remained the same after controlling for the main operatingsystems used by participants, all ps < .019.

Operating system differences Emoji evaluations were com-pared between the Android and iOS operating systems(Table 3). These results showed that iOS emoji were evaluatedas being more aesthetically appealing, familiar, clear, andmeaningful, all 5,000-sample bootstrapped ps ≤ .006. In con-trast, no differences between operating systemswere found forvisual complexity, valence, and arousal, all 5,000-samplebootstrapped ps ≥ .059.

Subjective rating norms

To define subjective rating norms, data was further coded andanalyzed by stimulus. For each stimulus, we calculated fre-quencies, means, standard deviations and confidence intervals(CIs) in each dimension (see Appendix 1 in the supplementarymaterial). On the basis of these results, stimuli were catego-rized as low, moderate, or high in each dimension (for asimilar procedure, see Prada et al., 2015). When the CI includ-ed the response scale midpoint (i.e., four) stimuli were con-sidered Bmoderate^ in a given dimension. Stimuli were cate-gorized as Blow^ when the upper bound of the CI was belowthe scale midpoint and as Bhigh^ when the lower bound of theCI was above the scale midpoint. In the case of valence, Blow^

Table 2 Evaluations of each dimension (means and standard deviations) for emoticons and emoji, for the total sample and for men and women, as wellas mean difference tests

Total Sample (N = 505) Men (n = 143) Women (n = 362) Difference Test for Gender

Stimulus/Dimension M SD M SD M SD t p

Emoticons

Aesthetic appeal 3.01a 1.13 3.09a 1.19 2.89a 1.11 0.91 .373

Familiarity 3.38a 1.30 3.29a 1.31 3.42a 1.30 –1.03 .306

Visual complexity 3.39a 1.25 3.35a 1.20 3.41a 1.27 –0.50 .612

Clarity 3.52a 1.20 3.48a 1.17 3.54a 1.21 –0.49 .622

Valence 3.96a 0.82 3.96a 0.83 3.95a 0.82 0.08 .941

Arousal 3.90a 0.81 3.95a 0.80 3.88a 0.81 0.88 .379

Meaningfulness 3.69a 1.22 3.62a 1.22 3.70a 1.23 –0.39 .699

Emoji

Aesthetic appeal 4.65b*** 1.04 4.55b*** 0.99 4.69b*** 1.05 –1.42 .141

Familiarity 4.86b*** 1.18 4.55b*** 1.14 4.99b*** 1.17 3.85 <.001

Visual complexity 3.52a 1.24 3.52a 1.14 3.53a 1.27 0.05 .965

Clarity 5.33b*** 0.93 5.15b*** 0.91 5.41b*** 0.93 –2.87 .003

Valence 4.08b* 0.85 4.01a 0.80 4.11b* 0.87 –1.20 .204

Arousal 4.84b*** 0.86 4.73b*** 0.84 4.88b*** 0.87 –1.73 .079

Meaningfulness 5.43b*** 0.85 5.29b*** 0.81 5.49b*** 0.86 –2.43 .015

Subscripts indicate 5,000-bootstrap-sample paired-sample t tests comparing emoticons and emoji on each evaluative dimension, by column. Differentsubscripts indicate significant differences: *** p < .001, * p < .050. p values for gender differences correspond to 5,000-bootstrap-sample paired-sample ttests.

398 Behav Res (2018) 50:392–405

means negative, Bmoderate^means neutral, and Bhigh^meanspositive. Figures 2 and 3 present summaries of this analysisfor emoticons and emoji separately.

As is shown in Fig. 2, the majority of emoticons werecategorized as being low in aesthetic appeal (76.47%), fa-miliarity (57.65%), and clarity (50.59%), and as beingmoderately arousing (55.29%). Moreover, the results showthat most emoticons were categorized as being low(48.24%) or moderate (44.71%) in complexity, and aslow (43.53%) or moderate (36.47%) in meaningfulness.Regarding valence, the emoticons were distributed acrossthe three levels: negative (42.35%), neutral (30.59%), andpositive (27.06%).

Figure 3 shows that the majority of emoji were categorizedas being highly familiar (58.82%), clear (79.08%), arousing(65.36%), and meaningful (88.24%). The results further showthat emoji were categorized as being high (49.02%) or mod-erate (45.10%) in aesthetic appeal, and moderate (54.25%) orlow (43.10%) in complexity. Note that the emoji were some-what polarized in their valence, being mostly categorized as

either negative (49.02%) or positive (42.48%) in this dimen-sion. Figure 4 depicts examples of emoticons and emoji foreach level of each evaluative dimension.

Correlations between dimensions

Overall, the results showed significant correlations betweenthe dimensions (see Table 4). For example, meaningfulnesswas strongly correlated with aesthetic appeal (r = .547), famil-iarity (r = .648), clarity (r = .743), and arousal (r = .506).Clarity was strongly associated with aesthetic appeal(r = .538) and familiarity (r = .704). Also aesthetic appealwas also strongly associated with familiarity (r = .556).

Analysis of attributed meaning/emotion

In addition to meaningfulness ratings, participants were askedto indicate the meaning or emotion attributed to each stimulus.Percentage of responses was computed considering the sam-ple size that evaluated a given stimulus. Two independentjudges coded the meaning/emotion attributed by the partici-pants to each symbol (for a similar strategy, see, e.g., Pradaet al., 2015). Synonyms (e.g., Bdon’t speak^ and Bsilence,^EmjAp31) and singular/plural forms (e.g., Bsmiles^ andBsmile,^ Emot1c) were included in the same category. Themeaning of 15 emoticons was not categorized due to a lowpercentage of responses (i.e., <25%). For example, from the42 participants that evaluated Emot32, only eight indicatedmeaning, from which two were categorized as Bsmile,^ twoas Bignore,^ and the remaining were uncategorized. Note thatthe sum of percentages of both categories does not necessarilyequals 100. For example, 48.4% of the valid responses forEmjAp47 were categorized as Bglad,^ 25.8% as Bupsidedown,^ whereas the remaining responses (n = 8) were hetero-geneous and therefore uncategorized (e.g., Bnormality,^Bsarcasm^).

Table 3 Evaluations of each dimension (means and standarddeviations) for Android and iOS emoji for the total sample, as well asmean difference tests

Android iOS Difference Test

Stimulus/Dimension M SD M SD t (504) p

Aesthetic appeal 4.45 1.22 4.77 1.11 6.38 <.001

Familiarity 4.43 1.53 5.05 1.22 10.40 <.001

Visual complexity 3.59 1.32 3.55 1.27 –0.89 .370

Clarity 5.12 1.12 5.30 0.99 3.70 <.001

Valence 3.85 1.01 3.95 0.91 1.89 .059

Arousal 4.74 0.98 4.77 0.95 0.69 .494

Meaningfulness 5.31 1.04 5.42 0.92 2.72 .006

Results of 5,000-bootstrap-sample paired-sample t tests comparingAndroid and iOS emoji on each evaluative dimension.

Fig. 2 Emoticon frequency distributions for each dimension level. For valence: low = negative, moderate = neutral, high = positive

Behav Res (2018) 50:392–405 399

The percentages of meaning responses varied between4.3% (Emot75) and 95.0% (Emot01a) for emoticons (M=49.9%, SD = 24.1); between 46.9% (EmjAn24) and 100%(e.g., EmjAn71) for Android emoji (M = 84.6%, SD = 11.9);and between 48.8% (EmjAp24) and 100% (e.g., EmjAp57)for iOS emoji (M = 86.9%, SD = 11.3). The percentages variedbetween 90.7% (EmjFb17) and 100% (e.g., EmjFb76) for theFacebook emoji (M = 95.7%, SD = 2.9), and between 74.4%(EmjPe86) and 97.8% (e.g., EmjPe88) for the Emojipediaemoji (M = 82.9%, SD = 10.8). Within each operating system,

results regarding the first category showed that, on average,participants agreed on the meanings of both the Android(64.95%) and iOS (66.78%) emoji.

A detailed discussion of the meaning or emotion attributedto each stimulus would be too extensive. The complete mean-ing analysis is presented in Appendix 2 in the supplementarymaterial alongside the Unicode intended meaning for compar-ison purposes. In some cases, the meaning categorization con-verged with the Unicode intended meaning. For instance, par-ticipants attributed a congruent meaning to the Bwinking face^

Fig. 3 Emoji frequency distributions for each dimension level. For valence: low = negative, moderate = neutral, high = positive

Fig. 4 Sample emoticons and emoji for each level across dimensions (LEED stimulus codes are included). For valence: low = negative, moderate =neutral, high = positive

400 Behav Res (2018) 50:392–405

stimulus in its different formats. For emoticon (Emot08a) themost frequent meanings were Bwink^ (40.5%) and Bagree^(21.6%), for the iOS emoji (EmjAp08) these were Bagree/compliance^ (40.0%) and Bwink^ (28.6%), and for theAndroid emoji (EmjAn08) these were Bwink^ (40.6%) andBcompliance^ (25.0%).

In other cases there was only partial convergence. For ex-ample, the emoji Bface savoring delicious food^ wasinterpreted as Bcheeky/fun^ (63.2%) and Btasty^ (18.4%) inthe iOS emoji (EmjAp10), and as Bwink/cheeky^ (59.4%) andBtasty^ (12.5%) in the Android emoji (EmjAn10). In anotherexample, the emoji Bimp^ was attributed the meanings Bevil^(60.0%) and Bmischief/prank^ (30.0%) in the iOS emoji(EmjAp70), and Bevil/mischief^ (62.5%) and Brage^(22.5%) in the Android emoji (EmjAn70).

For other stimuli, the attributed meaning differed acrossoperating systems and from the Unicode intended meaning.For example, the emoji Bdizzy face^ was attributed the mean-ing Bshocked^ (66.7%) in the iOS emoji (EmjAp66), andBconfusion^ (46.5%) and Bhypnotized^ (18.6%) in theAndroid emoji (EmjAn66). These examples clearly illustratethat the meaning participants assign to emoji is not alwaysconvergent with their Unicode intended meaning and alsovaries across operating systems.

Discussion

In this article we have presented the LEED, which includes238 emoticons and emoji, evaluated across seven evaluativedimensions: aesthetic appeal, familiarity, visual complexity,clarity, valence, arousal, and meaningfulness. Additionally,participants attributed meaning to each stimulus. To ourknowledge, this is the first available emoticon/emoji norma-tive database.

Results showed that, in comparison to emoticons, emoji areperceived as more aesthetically appealing, familiar, clear, andmeaningful. Most emoticons were categorized as low in aes-thetic appeal, familiarity, clarity, valence, and meaningfulness,whereas most emoji were categorized as high in familiarity,

clarity, arousal, and meaningfulness. This may be associatedwith an increasing popularity and use of emoji. Indeed, recentevidence shows that as emoji usage has increased the usage ofemoticons has decreased (Pavalanathan & Eisenstein, 2015).Furthermore, in the case of stimuli depicting facial cues, thegraphical representation of emoji may be more appealing be-cause they are better proxies to human facial expressions (e.g.,Ganster et al., 2012).

Results also showed no gender differences regarding theevaluation of emoticons. Emoji, however, were evaluated asmore familiar, clear, and meaningful by women. This findingconverges with empirical evidence showing that women aremore likely thanmen to use emoji (e.g., Fullwood, Orchard, &Floyd, 2013).

Recent literature has suggested the need to take into ac-count possible differences in emoji evaluation across operat-ing systems (Miller et al., 2016). Indeed, our results showedthat iOS emoji were evaluated asmore aesthetically appealing,familiar, clear and meaningful than Android emoji. We alsofound significant correlations between the evaluative dimen-sions (e.g., stimuli that were perceived as more meaningfulwere also perceived as more aesthetically appealing, familiar,clear and arousing). This pattern replicates findings from da-tabases of other visual stimuli using the same evaluative di-mensions (Garrido et al., 2016; Prada et al., 2015).

In addition to presenting normative ratings across dimen-sions, our database includes participants’ interpretation of themeaning of each stimulus. Participants were more likely toattribute meaning to emoji than to emoticons irrespectivelyof the operating system (iOS vs. Android). It is important tonote that even though participants described the meaning interms of what the stimulus directly represents (e.g., wink),they were also likely to go beyond this mere description andinfer its intent (e.g., being cheeky). This is particularly rele-vant because it allows researchers to assess the extent to whichthe intended meaning overlaps with the meaning attributed byusers, and more importantly because our findings show this isnot always the case. However, as in previous research, ourcoding system for the meaning has shortcomings that renderthis overlap subjective.

Table 4 Pearson’s correlations between the dimensions

Dimensions 1 2 3 4 5 6

1. Aesthetic appeal –

2. Familiarity .556*** –

3. Visual complexity –.038*** –.188*** –

4. Clarity .538*** .704*** –.175*** –

5. Valence .403*** .250*** –.032*** .176*** –

6. Arousal .266*** .314*** .106*** .398*** –.005 –

7. Meaningfulness .547*** .648*** –.062*** .743*** .123*** .506***

***Correlation is significant at the .001 level (two-tailed).

Behav Res (2018) 50:392–405 401

Emoticons and emoji are often analyzed in the absence ofinformation about the contexts in which they are communi-cated (Gaspar, Pedro, Panagiotopoulos, & Seibt, 2016). Thiswas also the case of the present research, in which ratings wereobtained by presenting the stimuli in isolation. This can con-stitute a limitation, because the interpretation of visual stimuliis often context-dependent (e.g., Wolff & Wogalter, 1998).Emoticons/emoji are typically incorporated in a message andresearch has already shown that they can influence how themessage is interpreted (e.g., Derks et al., 2008; Fullwoodet al., 2013). Moreover, the reverse may occur, such that thecontent of the message can influence the interpretation ofemoticons/emoji (e.g., Miller et al., 2016). For instance, awinking emoticons/emoji can be interpreted differently whenaccompanied by BLet’s go to the movies ;)^ versus BLet’swatch a movie at my place ;)^ Furthermore, emoticons/emoji interpretation can also depend on how the sender’sgoals are perceived (Gaspar, Barnett, & Seibt, 2015; Gasparet al., 2016). For instance, winking emoji accompanying asarcastic remark can be differently interpreted when the senderis a close friend or when the sender is one’s boss.

Another limitation to the present study concerns the specificcultural context in which this dataset was developed. Culture hasemerged as a factor that influences emoticon and emoji usage inonline communication (Park et al., 2014). Our normative datasetwas obtained with Portuguese participants and, according to re-cent data (Ljubešić & Fišer, 2016), Portugal ranks fourth inEurope for emoji usage on Twitter. Nevertheless, as with othernormative databases, generalizations to other populations shouldbe made with caution and cross-validation is recommended.Therefore, future studies should consider extending this databaseto other countries/cultures to assess cross-cultural differences andsimilarities. It should also be noted that differences may arisebetween studies that analyze how emoticon and emoji areevaluated in isolation from the context in which they are oftenused, and those focusing on how users actually contextualizethem in communication. For example, in our study participantsperceived emoji as negative or positive, whereas the work byNovak and colleagues (2015) showed that users mostly use pos-itive emoji in their tweets.

Finally, the results from the meaning analysis indicated thatintended meaning and users’ interpretation of that meaning donot always overlap. Two independent coders analyzed andcategorized the responses given by participants to each stim-ulus. Although this procedure is not exempt from bias, thelack of overlap constitutes an important indicator that the se-lection of emoji and emoticons to use in research or practiceshould be carefully conducted, on the basis of more objectivenormative data such as that reported in the LEED. Other pro-cedures could be used to determine users’ interpretation ofmeaning. For instance, researchers could use forced choicetasks (i.e., decide which emotion/meaning is expressed bythe stimuli; Vaiman, Wagner, Caicedo, & Pereno, 2017).

The LEEDmostly includes stimuli depicting graphical rep-resentations of faces. Research has shown that this type ofemoji is processed similarly to other human nonverbal infor-mation (e.g., voice and facial expression; Yuasa et al., 2011)and that emoji can be used to prime social presence (Tung &Deng, 2007). Therefore, our stimuli can be used in affectiveprocessing studies and as experimental primes.. Future studiescould also seek to expand our normative ratings to other emojirepresenting humans (e.g., bodily postures and activities).Considering that recently new emoji varying in age groupand skin tone were added to the available set in differentplatforms, it would be interesting to examine whether theyare suitable as stimulus materials in research designed to ex-amine topics such as person perception, intergroup relations,and social influence.

The LEED is a useful tool for researchers and practitioners(e.g., public health officials) interested in conducting researchwith naturalistic data (e.g., user-generated messages shared onsocial media platforms). It can also be used in a variety ofexperimental paradigms, particularly when the control ofstimuli characteristics is required. Instead of their selection,coding, and analysis of emoticons and emoji relying on ad hoccategorization and intended meaning, researchers and analystscan rely on the systematic normative ratings offered by theLEED.

This type of database also has the potential to be used inmore applied contexts comprising ICTs mediated writtencommunication, such as in marketing, education, and profes-sional contexts (e.g., Skiba, 2016; Skovholt et al., 2014).Particularly promising is the field of health informatics (see,e.g., Eysenbach, 2011). Both human-based and computer-based evaluations of ICT users reactions to health relatedevents have been used for a variety of public health issuesmonitoring and surveillance (e.g., influenza like diseases anddengue; Milinovich, Williams, Clements, & Hu, 2014). Insuch monitoring, machine-learning algorithms and semanticlexicons often use computer-based techniques. These tech-niques would benefit if they were based on normative ratingssuch as those offered by the LEED.

Author note Part of this research was funded by grants from theFundação para a Ciência e Tecnologia awarded to the first (SFRH/BPD/73528/2010), third (UID/PSI/04810/2013), and fourth (PTDC/MHC-PCN/5217/2014) authors, and by a Marie Curie fellowship (FP7-PEOPLE-2013-CIG/631673) awarded to the fourth author. We thankNuno Porto for his assistance in preparing the figures.

References

Antheunis, M. L., Valkenburg, P. M., & Peter, J. (2007). Computer-mediated communication and interpersonal attraction: An experi-mental test of two explanatory hypotheses. Cyber Psychology andBehavior, 10, 831–836. doi:10.1089/cpb.2007.9945

402 Behav Res (2018) 50:392–405

Antheunis, M. L., Valkenburg, P. M., & Peter, J. (2010). Gettingacquainted through social network sites: Testing a model of onlineuncertainty reduction and social attraction. Computers in HumanBehavior, 26, 100–109. doi:10.1016/j.chb.2009.07.005

Beltrone, G. (2015). Everyone is an emoji in this bizarre and terrifyingFrench McDonald’s ad. AdWeek. Retrieved from www.adweek.com/adfreak/everyone-emoji-bizarre-and-terrifying-french-mcdonalds-ad-166335

Blechert, J., Meule, A., Busch, N. A., & Ohla, K. (2014). Food-pics: Animage database for experimental research on eating and appetite.Frontiers in Psychology, 5, 617. doi:10.3389/fpsyg.2014.00617

Bonin, P., Peereman, R., Malardier, N., Méot, A., & Chalard, M. (2003).A new set of 299 pictures for psycholinguistic studies: French normsfor name agreement, image agreement, conceptual familiarity, visualcomplexity, image variability, age of acquisition, and naming laten-cies. Behavior Research Methods, Instruments, & Computers, 35,158–167. doi:10.3758/BF03195507

Bradley, M.M., & Lang, P. J. (1999a). Affective Norms for English Words(ANEW): stimuli, instruction manual, and affective ratings(Technical Report C-1). Gainesville: University of Florida, Centerfor Research in Psychophysiology.

Bradley, M. M., & Lang, P. J. (1999b). International Affective DigitizedSounds (IADS): Stimuli, instruction manual and affective ratings(Technical Report B-2). Gainesville: University of Florida, Centerfor Research in Psychophysiology.

Burnap, P., Gibson, R., Sloan, L., Southern, R., & Williams, M. (2016).140 characters to victory? using twitter to predict the UK 2015general election. Electoral Studies, 41, 230–233. doi:10.1016/j.electstud.2015.11.017

Carvalho, P., Sarmento, L., Silva, M. J., & de Oliveira, E. (2009). Cluesfor detecting irony in user-generated contents: Oh…‼ It’s Bso easy^;-). In Proceedings of the 1st international CIKMworkshop on topic–sentiment analysis for mass opinion (pp. 53–56). New York: ACMPress. doi:10.1145/1651461.1651471

Chambers, C. T., & Craig, K. D. (1998). An intrusive impact of anchors inchildren’s faces pain scales. Pain, 78, 27–37. doi:10.1016/S0304-3959(98)00112-2

Charbonnier, L., van Meer, F., van der Laan, L. N., Viergever, M. A., &Smeets, P. A. M. (2016). Standardized food images: Aphotographing protocol and image database. Appetite, 96, 166–173. doi:10.1016/j.appet.2015.08.041

Comesaña, M., Soares, A. P., Perea, M., Piñeiro, A. P., Fraga, I., &Pinheiro, A. (2013). ERP correlates of masked affective primingwith emoticons. Computers in Human Behavior, 29, 588–595. doi:10.1016/j.chb.2012.10.020

Dan-Glauser, E. S., & Scherer, K. R. (2011). The Geneva AffectivePicture Database (GAPED): A new 730-picture database focusingon valence and normative significance.Behavior ResearchMethods,43, 468. doi:10.3758/s13428-011-0064-1

Davidov, D., Tsur, O., & Rappoport, A. (2010). Enhanced sentimentlearning using Twitter hashtags and smileys. In Proceedings of the23rd international conference on computational linguistics (pp.241–249). Stroudsburg: Association for ComputationalLinguistics. Retrieved from http://dl.acm.org/citation.cfm?id=1944566.1944594

Derks, D., Bos, A. E. R., & von Grumbkow, J. (2008). Emoticons andonline message interpretation. Social Science Computer Review, 26,379–388. doi:10.1177/0894439307311611

Dresner, E., & Herring, S. C. (2010). Functions of the nonverbal in CMC:Emoticons and illocutionary force. Communication Theory, 20,249–268. doi:10.1111/j.1468-2885.2010.01362.x

Ebner, N. C., Riediger, M., & Lindenberger, U. (2010). FACES—A da-tabase of facial expressions in young, middle-aged, and older wom-en and men: Development and validation. Behavior ResearchMethods, 42, 351–362. doi:10.3758/BRM.42.1.351

Eysenbach, G. (2011). Infodemiology and infoveillance: Tracking onlinehealth information and cyberbehavior for public health. AmericanJournal of Preventive Medicine, 40, S154–S158. doi:10.1016/j.amepre.2011.02.006

Fane, J., MacDougall, C., Jovanovic, J., Redmond, G., & Gibbs, L.(2016). Exploring the use of emoji as a visual research method foreliciting young children’s voices in childhood research. Early ChildDevelopment and Care. doi:10.1080/03004430.2016.1219730

Fullwood, C., Orchard, L. J., & Floyd, S. A. (2013). Emoticon conver-gence in internet chat rooms. Social Semiotics, 23, 648–662. doi:10.1080/10350330.2012.739000

Ganster, T., Eimler, S. C., & Krämer, N. C. (2012). Same same butdifferent!? the differential influence of smilies and emoticons onperson perception. Cyber Psychology, Behavior, and SocialNetworking, 15, 226–230. doi:10.1089/cyber.2011.0179

Garcia-Marques, T., Mackie, D.M., Claypool, H.M., & Garcia-Marques,L. (2004). Positivity can cue familiarity. Personality and SocialPsychology Bulletin, 30, 585–593. doi:10.1177/0146167203262856

Garrido, M. V., Lopes, D., Prada, M., Rodrigues, D., Jerónimo, R., &Mourão, R. P. (2016). The many faces of a face: Comparing stillsand videos of facial expressions in eight dimensions (SAVE data-base). Behavior Research Methods. doi:10.3758/s13428-016-0790-5

Gaspar, R., Barnett, J., & Seibt, B. (2015). Crisis as seen by the individ-ual: The norm deviation approach. Psyecology, 6, 103–135. doi:10.1080/21711976.2014.1002205

Gaspar, R., Pedro, C., Panagiotopoulos, P., & Seibt, B. (2016). Beyondpositive or negative: Qualitative sentiment analysis of social mediareactions to unexpected stressful events. Computers in HumanBehavior, 56, 179–191. doi:10.1016/j.chb.2015.11.040

Gülşen, T. T. (2016). You tell me in emojis. In O. Ogata & T. Akimoto(Eds.),Computational and cognitive approaches to narratology (pp.354–375). Hershey: Information Science Reference.

Han, D. H., Yoo, H. J., Kim, B. N., McMahon, W., & Renshaw, P. F.(2014). Brain activity of adolescents with high functioning autism inresponse to emotional words and facial emoticons. PLoS ONE, 9, 1–8. doi:10.1371/journal.pone.0091214

Hogenboom, A., Bal, D., Frasincar, F., Bal, M., de Jong, F., & Kaymak,U. (2013). Exploiting emoticons in sentiment analysis. InProceedings of the 28th annual ACM symposium on appliedcomputing (pp. 703–710). New York: ACM Press. doi:10.1145/2480362.2480498

Huang, A. H., Yen, D. C., & Zhang, X. (2008). Exploring the potentialeffects of emoticons. Information Management, 45, 466–473. doi:10.1016/j.im.2008.07.001

Jaeger, S. R., Vidal, L., Kam, K., & Ares, G. (2017). Can emojibe used as a direct method to measure emotional associationsto food names? Preliminary investigations with consumers inUSA and China. Food Quality and Preference, 56, 38–48.doi:10.1016/j.foodqual.2016.09.005

Kaye, L. K., Wall, H. J., & Malone, S. A. (2016). BTurn that frownupside-down^: A contextual account of emoticon usage on differentvirtual platforms. Computers in Human Behavior, 60, 463–467. doi:10.1016/j.chb.2016.02.088

Kerkhof, I., Goesaert, E., Dirikx, T., Vansteenwegen, D., Baeyens, F.,D’Hooge, R., & Hermans, D. (2009). Assessing valence indirectlyand online. Cognition and Emotion, 23, 1615–1629. doi:10.1080/02699930802469239

Krohn, F. B. (2004). A generational approach to using emoticons asnonverbal communication. Journal of Technical Writing andCommunication, 34, 321–328. doi:10.2190/9eqh-de81-cwg1-qll9

Lang, P. J., Bradley, M. M., & Cuthbert, B. N. (2008). InternationalAffective Picture System (IAPS): Affective ratings of pictures andinstruction manual (Technical Report A-8). Gainesville: Universityof Florida.

Behav Res (2018) 50:392–405 403

Lea, M., & Spears, R. (1992). Paralanguage and social perception incomputer‐mediated communication. Journal of OrganizationalCommunication, 2, 321–341. doi:10.1080/10919399209540190

Liu, B. (2012). Sentiment analysis and opinion mining. San Rafael:Morgan & Claypool.

Liu, K.-L., Li, W.-J., & Guo, M. (2012). Emoticon smoothed languagemodels for twitter sentiment analysis. In Proceedings of the 26thAAAI conference on artificial intelligence (pp. 1678–1684). NewYork: AAAI Press.

Ljubešić, N., & Fišer, D. (2016). A global analysis of emoji usage. Inproceedings of the 10th Web as corpus workshop (WAC-X) and theEmpiriST shared task (pp. 82–89). Stroudsburg: Association forComputational Linguistics.

Lo, S.-K. (2008). The nonverbal communication functions of emoticonsin computer-mediated communication. Cyber Psychology andBehavior, 11, 595–597. doi:10.1089/cpb.2007.0132

McDougall, S. J. P., Curry, M. B., & de Bruijn, O. (1999). Measuringsymbol and icon characteristics: Norms for concreteness, complex-ity, meaningfulness, familiarity, and semantic distance for 239 sym-bols. Behavior Research Methods, Instruments, & Computers, 31,487–519. doi:10.3758/BF03200730

Mendonça, R., Garrido, M. V., & Semin, G. R. (2016). A standardizeddatabase for Portuguese faces looking towards left, right and thefront: The LRF Face Corpus. Manuscript submitted for publication.

Milinovich, G. J., Williams, G.M., Clements, A. C. A., & Hu,W. (2014).Internet-based surveillance systems for monitoring emerging infec-tious diseases. Lancet Infectious Diseases, 14, 160–168. doi:10.1016/S1473-3099(13)70244-5

Miller, H., Thebault-Spieker, J., Chang, S., Johnson, I., Terveen, L., &Hecht, B. (2016). BBlissfully happy^ or Bready to fight^: varyinginterpretations of emoji. In International Conference on Web andSocial Media (ICWSM) ’16 (pp. 259–268). New York: AAAI Press.

Moore, A., Steiner, C. M., & Conlan, O. (2013). Design and developmentof an empirical smiley-based affective instrument. In 1st workshopon emotions and personality in personalized services (pp. 24–30).Rome: UMAP.

Neff, J. (2015). Dove launches curly-haired emojis to end straight-hairdominance. Advertising Age. Retrieved from http://adage.com/article/digital/dove-launches-curly-haired-emojis-address-void/301203/

Negishi, M. (2014). Meet Shigetaka Kurita, the father of emoji. WallStreet Journal. Retrieved from http://blogs.wsj.com/japanrealtime/2014/03/26/meet-shigetaka-kurita-the-father-of-emoji/

Nelson, R. A., Tossell, C. C., & Kortum, P. (2015). Emoticon use inmobile communications :-). In Z. Yan (Ed.), Encyclopedia of mobilephone behavior (pp. 1–11). Hershey: IGI Global.

Novak, P. K., Smailović, J., Sluban, B., & Mozetič, I. (2015). Sentimentof emojis. PLoS ONE, 10, e0144296. doi:10.1371/journal.pone.0144296

Pang, B., & Lee, L. (2008). Opinion mining and sentiment analysis.Foundations and Trends in Information Retrieval, 2, 1–135. doi:10.1561/1500000011

Park, J., Baek, Y. M., & Cha, M. (2014). Cross-cultural comparison ofnonverbal cues in emoticons on twitter: Evidence from big dataanalysis. Journal of Communication, 64, 333–354. doi:10.1111/jcom.12086

Paul, M. J., & Dredze, M. (2011). You are what you tweet: analyzingtwitter for public health. In Fifth international AAAI conference onweblogs and social media (pp. 265–272). New York: AAAI Press.

Pavalanathan, U., & Eisenstein, J. (2015). Emoticons vs. emojis onTwitter: A causal inference approach. arXiv:1510.08480 [cs.CL].

PEW Research Center. (2016). Smartphone ownership and Internet us-age continues to climb in emerging economies. Retrieved fromwww.pewglobal.org/2016/02/22/smartphone-ownership-and-internet-usage-continues-to-climb-in-emerging-economies/

Prada, M., Rodrigues, D., Silva, R. R., & Garrido, M. V. (2015). LisbonSymbol Database (LSD): Subjective norms for 600 symbols.Behavior Research Methods, 48, 1370–1382. doi:10.3758/s13428-015-0643-7

Proctor, R. W., & Vu, K.-P. L. (1999). Index of norms and ratings pub-lished in the psychonomic society journals. Behavior ResearchMethods, Instruments, & Computers, 31, 659–667. doi:10.3758/BF03200742

Richmond, V., & McCroskey, J. (2009). Human communication theoryand research: Traditions and models. In D. Stacks & M. Salwen(Eds.), An integrated approach to communication theory andresearch (2nd ed., pp. 223–231). New York: Routledge.

Siegel, R. M., Anneken, A., Duffy, C., Simmons, K., Hudgens, M.,Lockhart, M. K., & Shelly, J. (2015). Emoticon use increases plainmilk and vegetable purchase in a school cafeteria without adverselyaffecting total milk purchase.Clinical Therapeutics, 37, 1938–1943.doi:10.1016/j.clinthera.2015.07.016

Skiba, D. J. (2016). Face with tears of joy is word of the year: Are emoji asign of things to come in health care? Nursing EducationPerspectives, 37, 56–57. doi:10.1097/01.NEP.0000476112.24899.a1

Skovholt, K., Grønning, A., & Kankaanranta, A. (2014). The communi-cative functions of emoticons in workplace e-mails: :-). Journal ofComputer-Mediated Communication, 19, 780–797. doi:10.1111/jcc4.12063

Snodgrass, J. G., & Vanderwart, M. (1980). A standardized set of 260pictures: Norms for name agreement, image agreement, familiarity,and visual complexity. Journal of Experimental Psychology:Human Learning and Memory, 6, 174–215. doi:10.1037/0278-7393.6.2.174

Thelwall, M., Buckley, K., Paltoglou, G., Cai, D., & Kappas, A. (2010).Sentiment strength detection in short informal text. Journal of theAmerican Society for Information Science and Technology, 61,2544–2558. doi:10.1002/asi.21416

Thelwall, M., Buckley, K., & Paltoglou, G. (2012). Sentiment strength detec-tion for the social web. Journal of the American Society for InformationScience and Technology, 63, 163–173. doi:10.1002/asi.21662

Thompson, D., & Filik, R. (2016). Sarcasm in written communication:Emoticons are efficient markers of intention. Journal of Computer-Mediated Communication, 21, 105–120. doi:10.1111/jcc4.12156

Tossell, C. C., Kortum, P., Shepard, C., Barg-Walkow, L. H., Rahmati, A.,& Zhong, L. (2012). A longitudinal study of emoticon use in textmessaging from smartphones. Computers in Human Behavior, 28,659–663. doi:10.1016/j.chb.2011.11.012

Tung, F.-W., & Deng, Y.-S. (2007). Increasing social presence of socialactors in e-learning environments: Effects of dynamic and staticemoticons on children. Displays, 28, 174–180. doi:10.1016/j.displa.2007.06.005

Vaiman, M., Wagner, M. A., Caicedo, E., & Pereno, G. L. (2017).Development and validation of an argentine set of facial expressionsof emotion. Cognition and Emotion, 31, 249–260. doi:10.1080/02699931.2015.1098590

Vashisht, G., & Thakur, S. (2014). Facebook as a corpus for emoticons-based sentiment analysis. International Journal of EmergingTechnology and Advanced Engineering, 4, 904–908.

Vidal, L., Ares, G., & Jaeger, S. R. (2016). Use of emoticon and emoji intweets for food-related emotional expression. Food Quality andPreference, 49, 119–128. doi:10.1016/j.foodqual.2015.12.002

Walther, J. B. (1996). Computer-mediated communication: Impersonal,interpersonal, and hyperpersonal interaction. CommunicationResearch, 23, 3–43. doi:10.1177/009365096023001001

Walther, J. B., & D’Addario, K. P. (2001). The impacts of emoticons onmessage interpretation in computer-mediated communication.Social Science Computer Review, 19, 324. doi:10.1177/089443930101900307

Wang, H., & Castanon, J. A. (2015). Sentiment expression via emoticonson social media. Retrieved from arXiv:1511.02556

404 Behav Res (2018) 50:392–405

Wang, W., Zhao, Y., Qiu, L., & Zhu, Y. (2014). Effects of emoticons onthe acceptance of negative feedback in computer-mediated commu-nication. Journal of the Association for Information Systems, 15(8),3. Retrieved from http://aisel.aisnet.org/jais/vol15/iss8/3

Wiebe, J., Wilson, T., & Cardie, C. (2005). Annotating expressions ofopinions and emotions in language. Language Resources andEvaluation, 39, 165–210. doi:10.1007/s10579-005-7880-9

Wohl, J. (2016, August 16). Howmarketers can win the great emoji armsrace. Advertising Age. Retrieved from http://adage.com/article/digital/marketers-emoji-arms-race/303361/

Wolff, J. S., & Wogalter, M. S. (1998). Comprehension of pictorial sym-bols: Effects of context and test method. Human Factors, 40, 173–186. doi:10.1518/001872098779480433

Yuasa, M., Saito, K., & Mukawa, N. (2011). Brain activity associatedwith graphic emoticons. The effect of abstract faces in communica-tion over a computer network. Electrical Engineering in Japan, 177,36–45. doi:10.1002/eej.21162

Behav Res (2018) 50:392–405 405