AUX and UX Evaluation of User Tools in SocialNetworks

Luis Martın Sanchez-AdameComputer Science Department

CINVESTAV-IPNMexico City, Mexico

luismartin.sanchez@cinvestav.mx

Sonia Mendoza∗Computer Science Department

CINVESTAV-IPNMexico City, Mexico

smendoza@cs.cinvestav.mx

Beatriz A. Gonzalez BeltranSystems DepartmentUAM-Azcapotzalco

Mexico City, Mexicobgonzalez@azc.uam.mx

Jose RodrıguezComputer Science Department

CINVESTAV-IPNMexico City, Mexico

rodriguez@cs.cinvestav.mx

Amilcar Meneses ViverosComputer Science Department

CINVESTAV-IPNMexico City, Mexico

ameneses@cs.cinvestav.mx

Abstract—Online social networks provide technological sup-port for making up virtual groups of any size, whose membersshare common interests and interact through the Internet. Aperson joins a social network not only owing to its popularity andthe quality of its contents, but also thanks to the User Experience(UX) that the platform offers. A crucial factor in the growth andsurvival of any virtual social group is participation, which shouldbe facilitated by suitable user tools supplied by the platform.However, this is not always the case. Anticipated User eXperience(AUX) allows knowing the idealisations, hopes, and desires of theusers in a very early stage of any development. In this paper,we propose an AUX and UX evaluation method for user toolsof social networks, whose goal is to improve their design, andthrough it, the participation of people. We tested our method byan experimental design that included the construction of paperprototypes and execution of tasks in actual platforms. The AUXand UX of the participants were measured with AttrakDiff.As shown by our results, based on their previous experience,participants have fixed ideas on the behaviour of user tools,having a visible impact when their expectations are not met.

Index Terms—User Experience, Anticipated User Experience,Social Networks, Participation, User Tools

I. INTRODUCTION

In Computer Science, social networks, also called onlinecommunities or e-communities, are an area of interest. In thescientific literature, we can encounter many definitions of theterm “social network” [1], [2], [3], [4]. Nevertheless, we chosethe one proposed by Lee at al. [5]: “cyberspace(s) supportedby computer-based information technology, centred upon com-munication and interaction of participants to generate member-driven content, resulting in a relationship being built”, as wethink this expresses the complexity of social networks.

Shneiderman et al. [6] present challenges that Human-Computer Interaction researchers have to face in the years tocome. Those problems are the product of human populationgrowth and its consequences. One of these problems states

∗Corresponding author.

that it is needed to shift from user experience to communityexperience. Many research fields (e.g., Sociology, Psychology,and Economics) are advocated for the resolution of theseproblems in social networks. However, as this is a broadsubject, a multidisciplinary approach seems to be natural.

In the design of successful social networks, sociability andusability are key factors. Sociability concerns the relationshipestablished among the members, and usability refers to howthey interact with technology [7], [8].

A logical consequence of sociability is participation, aninherent issue of social networks. Consequently, there aremany studies to comprehend why people participate in asocial network [9], [10], [11], [12]. Participation is crucial,as the success of any social network strongly depends on1) the cooperation among its users to generate content andcontribute to the community [13], and 2) the user interactionwith businesses, organisations, peers, relatives, and friendsto co-create their consumption experience and satisfy theirneeds [14], [15], [16].

Dealing with those interactions is a design task, so UXoffers a holistic view to addressing this endeavour. UX ischaracterised by hedonic and pragmatic factors. Hedonic fac-tors include all users’ perceptions and responses such as emo-tions, beliefs, preferences, behaviours, and accomplishmentsresulting from the actual and anticipated use of a product,system or service. Pragmatic factors are a consequence of thecharacteristics of the system subject to evaluation: usability,functionality, performance, interactive behaviour, and assistivecapabilities [17].

AUX deals with experiences and feelings of what theuser expects to occur when imagining using an interactiveproduct or system [18]. The purpose of AUX evaluation isto identify whether a given concept can provide the kind ofUX anticipated by developers for its future users [19]. Thestudy of AUX has been proven valuable, although few jobshave explored it [20], [21], [22], [23].

104

2018 IEEE/WIC/ACM International Conference on Web Intelligence (WI)

978-1-5386-7325-6/18/$31.00 ©2018 IEEEDOI 10.1109/WI.2018.0-101

TABLE IRELATED WORK ANALYSIS

Work Key facts AUX evaluation Shortcomings Fields

Kukka et al. [24]Addition of social network informa-tion within 3D environments.

Paper prototypes. Just social media tools studied. 3D environments.

Magin et al. [25]Identification of factors for negativeUX.

Questionnaires. Preliminary study. Mobile apps.

Margetis et al. [26]Unobtrusive augmented reality sys-tem for physical books.

Questionnaires. Absence of end-users evaluation. Augmented reality.

Moser et al. [27] Envisioned gameplay ideas. Workshops. Lack of generalisations. Games.Sato et al. [28] Review of agent systems in CoP. None. UX analysis are not presented. CoP.

Wurhofer et al. [29] Drivers’ UX over time. Interviews.Very demanding study process forparticipants.

Driving UX.

Zhang et al. [30]Narrative grounding to improve learn-ing of foreign languages.

Paper prototypes. No results overtime. Language learning.

In this article, we propose a novel AUX and UX evaluationmethod through which developers can obtain a knowledge baseto create their designs. Such knowledge would have as source:opinions, desires, experience, and idealisations of users. As aresult, user tools can have a positive impact on the participationof users in social networks. Using our method, developershave the opportunity to focus on the nuances they want of theinteraction and communication tools of social networks, whichwould not happen if they decided to use a general-purposeAUX and UX methods. To test our proposal, we constructedan experimental design that involves the creation of paperprototypes for the accomplishment of three tasks and the actualexecution of these tasks on YouTube, Facebook, and Reddit.Using the AttrakDiff tool, we measured the AUX and UX of20 participants. The results revealed that participants sharedfirm convictions about how user tools should behave, basedon their tastes and prior knowledge. The impact is evidentwhen these expectations were not met.

This paper is organised as follows. After analysing relatedwork, in Section III we present the research methodology usedto develop our proposal. Then, in Section IV, we describe ourevaluation method, and in Section V, we present the resultsobtained from the tests conducted on users of social networks.Finally, in Section VI, we conclude this work and describesome ideas of future work.

II. RELATED WORK

In this section, we present an overview of previous studieson AUX, emphasising the lessons learned.

Kukka et al. [24] gathered information and requirements forthe integration of social media content into 3D environments.From their AUX evaluation, they identified issues and guide-lines that designers of 3D systems should consider. This is apreliminary study that only involves social networks.

Magin et al. [25] identified aspects that can cause neg-ative UX in mobile applications. They experimented witha functional prototype, contrasting the UX before and afterthe participants used the application. The results showed thatnegative emotions are experienced in case of poor usability.

Margetis et al. [26] developed “SESIL”, a system targetedto enhance reading and writing activities on physical books

through unobtrusive monitoring of users. Although they madean expert evaluation, it is not clear whether they are actuallymeasuring AUX or just UX after having used the prototype.

Moser et al. [27] captured the envisioned gameplay ideasproposed by children around the world, through a series ofworkshops. Although they describe some discoveries that theirresearch provided, they do not explicitly present critical pointsthat can be developed later.

Sato et al. [28] presented a review of agent and multi-agent systems containing features that could be used to supportCommunities of Practice (CoP). Although the impact of agentapplications on tasks inside communities is clear, the researchdoes not include UX evaluations.

Wurhofer et al. [29] investigated the relationship betweenpredicted and actual driving experiences. They found thatmany factors influence commuters’ predictions, such as moodand the time of day. This was a study on UX over time, butwithout any Graphical User Interface (GUI) involved.

Zhang et al. [30] presented the design process and findingsfrom a storytelling game to help classroom teachers to encour-age practising creative foreign language production. Through agame with paper cards, the authors describe a series of designrecommendations for language learning systems. However,they do not present guidelines that can be applied directlyto a GUI.

These works show the value of AUX and how it can beuseful to predict design problems. However, none exploreswhether the different user tools of a social network need to beimproved to encourage participation. Table I shows a compar-ison between the works already described; it resumes criticalfacts, how AUX evaluation was carried out, shortcomings, andthe development context.

III. RESEARCH METHODOLOGY

The research methodology for the development of ourevaluation method is based on the Design Science ResearchMethodology (DSRM) process model proposed by Peffers etal. [31]. We chose this methodology because its popularityin the state of the art, and it has proved useful in similarproblems [13], [32], [33].

105

An Objective-Centred Initiation has been chosen as a re-search entry point because our goal is to improve participationin social networks. As for Identification & Motivation, we havealready described the importance of participation for socialnetworks and the role of UX in that matter. The Objective ofa Solution, the second step of the process, is to develop anevaluation method that helps developers to create user toolsto improve user interaction, thus encouraging participation.The third step Design & Development is the description ofour method (see Section IV). Demonstration and Evaluationare the trials that we performed and the results we obtainedrespectively (see Section V). This is the first iteration of theprocess. Subsequent iterations will begin in the Design &Development stage, in order to improve the evaluation method.

IV. EVALUATION METHOD

According to Koh et al. [34], communities face challengesfrom the social perspective (communication, motivation, andleadership) and the practical perspective (computer technol-ogy). We consider user tools as the intersection of bothperspectives (see Fig. 1). User tools are a set of widgets ofa social network platform that allow users to exchange infor-mation and interact with each other, e.g., chats, commentingsystems, avatars, recommender systems, and “like” buttons.The granularity level is by tasks, i.e., if the set of widgetsallows the user to perform a specific task, then it is a user tool.User tools are part of the platform, a computational supportthat is crucial to the success of a social network [35], [36],[37], [38].

Community platform -Technology

Users -Communication -Motivation -Leadership

UserTools

Participation

AUXPragmaticFactors

HedonicFactors

Fig. 1. We aim at improving user tools through AUX, thus increasingparticipation.

We want to evaluate the impact that these user tools havein users’ activities within social networks, i.e., to what degreethe UX provided by the widgets of a social network affectsparticipation. For achieving this goal, we propose an evaluationmethod organised in six steps (see Fig. 2).

Next, we explain each step of the method. We also proposethe example “one user wants to stablish a conversation withanother user” that is developed at each stage:

∙ Set Goals: It refers to the objectives that developerswant to reach. For example, that users achieve a remoteconversation as effective as if they were face to face.

∙ Identify Tasks: This step involves defining the stages thatthe user needs to do in order to achieve the previouslydefined goals. For example, the user can identify therecipient of the message, open the window or option ofdirect messages, write the message, and send it.

0201

03

0405

06

SetGoals

IdentifyTasks

IdentifyUser Tools

CompareResults

EvaluateAUX

EvaluateUX

Fig. 2. AUX evaluation method for social networks.

∙ Identify User Tools: This point concerns establishingwhich user tools are involved in the tasks already identi-fied. For example, avatars, user profiles, commands, andchat systems.

∙ Evaluate AUX: It consists of conducting an AUX evalua-tion. We use the criteria described by Yogasara et al. [18]:

– Intended Use: This criterion refers to the usagepurpose of each user tool. For example, the way theuser believes that a chat should work.

– Positive Anticipated Emotion: It concerns pleasantemotions that a user anticipates experiencing as aconsequence of interacting with a user tool.

– Desired Product Characteristics: We adapt to ourcontext the principles proposed by Morville [39],which state that a user tool has to be valuable, usable,useful, desirable, accessible, credible, and findable.

– User Characteristics: It has to do with the physicaland mental characteristics of users. For example,designing for children is not the same as for seniors.

– Experiential Knowledge: This criterion requiresknowing the experience that a user has. Is importantsince users need their prior knowledge to obtaininformation and make comparisons. For example, auser may ask whether a new system is better thananother already known.

– Favourable Existing Characteristics: It is related tothe characteristics that users have already identifiedas positive in similar tools. For example, a usercan say that they like to use the chat from anotherplatform, as it is fast, reliable, and easy to use.

∙ Evaluate UX: This step consists in conducting a UXevaluation. It can be as complicated as necessary, depend-ing on the criteria and tools that will be used. However,it is essential to evaluate all the aspects considered in theAUX evaluation so that the next point is achievable.

∙ Compare Results: Once both evaluations of UX werecarried out (anticipated and current), the results obtainedmust be compared. This contrast will allow developersto make decisions about the design of user tools. Forexample, juxtaposing what is idealised by the users with

106

a) b)Message

User 1

Title

Description of the problem

Code

Run

#import

Fig. 3. In preliminary tests, we only provided blank sheets and drawing materials; invariably we obtained prototypes similar to (a). Thus we decided to givethe participants sheets with a printed wireframe so they would use it as a canvas (b).

reality, checking whether their suggestions came to beimplemented or not.

Gathering the expectations and idealisations of users willallow developers to know the underlying needs, the possiblecontexts of use, and what users consider essential to obtain apleasant experience (AUX). Then, when measuring the actualUX of user tools, the results can be contrasted, e.g., if all theparticipants agreed that adding a specific characteristic wouldbe good, after implementing it and measuring it, was it indeedpositive? If something was implemented as the users describedit, but it was poorly qualified, what happened? Our evaluationmethod helps to answer those kinds of questions.

V. STUDY

Having already explained our evaluation method, the objec-tive of this study is to test it. The AUX and UX evaluation ofuser tools were performed in a quiet environment, to preventexternal sources from influencing the research. Each partici-pant individually took part in the study, under the supervisionof a moderator who sat next to the participant.

We had 20 participants, of which five were female. Theaverage age was 28.15 (max: 38, min: 20). First, participantscompleted a background questionnaire giving their demo-graphic information and prior experience with social networks,and then they did the tasks and evaluations. Each session lastedapproximately 40 minutes.

The participants were chosen based on their experience us-ing social networks. We believe that people with no experiencein these platforms could limit their ability to solve the giventasks, which could negatively influence this investigation.Besides, we think that we would receive better results if theparticipants had particular skills in social networks. We alsodecided to limit their age, using a range of 20 to 40 years, sothat the results were not influenced by users with particularrequirements.

The following sections describe, first, the methodology ofthe experiments (see Section V-A), and the results obtained(see Section V-B).

A. Following up the method

First, we stated that our goals are to test the ways of basicparticipation in a social network. For this, we define threetasks:

1) Send a private message from one user to another.2) Create a new publication.3) Search for a specific person or topic.

To identify the necessary user tools for each task, wethought of several alternatives. However, we believed that ifwe offered a predefined set to the participants, they would bebiased, since they would only use the tools that we provided,limiting significantly their feedback. So, the best choice is foreach participant to define their own user tools.

The next stage is the AUX and UX evaluation of user tools.For this, we ask participants to complete the following stages:

∙ Prototype construction: We asked participants to imag-ine that they were Web designers and that they were goingto build a new social network. Then, based on their expe-rience, they had to develop a prototype to solve a giventask (those mentioned above). To do this, participants hadpencils, pens, and sheets with a printed wireframe at theirdisposal (see Fig. 3). Once they finished developing andexplaining each prototype, participants had to evaluatetheir work through a questionnaire.Thus, this stage served for the participants to express theirideas of how they imagine the functionality of the GUI,what they believe best, and the user tools necessary tosolve a task. In this way, we evaluate the user tools AUX.

∙ Tasks with social networks: Once the three prototypesand their corresponding evaluations were finished, weasked the participants to perform the same tasks, but thistime on real platforms, in such a way that in Reddit theysent a private message, in Facebook they created a newpublication, and in YouTube, they made the search of aspecific channel. Similarly, after completing each task,they had to evaluate it through a questionnaire.We chose those three social networks due to the results ofthe background questionnaires. YouTube turned out to be

107

the most used platform, Facebook as the moderately used,and Reddit as the least used. In this way, we evaluate theuser tools UX.

Thus, each participant had to fill out six questionnaires,which correspond to three for evaluate AUX and three forUX, in such a way that only the final staging of our methodremains the comparison of results.

Fig. 4. Reddit obtained rather low grades in both dimensions, while theprototypes are located in the region “task-oriented” meaning that there isroom for improvement. Therefore, Reddit user tools did not precisely meetthe expectations of participants.

Fig. 5. Although quite close, Facebook obtained better results than theprototypes. In both cases, changes would have to be made to arrive at the“desired” region.

Fig. 6. In the search task, both evaluations are in the “task-oriented” region.However, YouTube got slightly better results.

B. Results

We decided to use the AttrakDiff1 tool [40] as it providesinformation about the level of specific UX dimensions in ourinterest (hedonic and pragmatic ones) in just one questionnaire.With the help of semantic differentials, users can indicate theirperception of a product. It is not a nuisance for the participantsas it is quick to implement and fill out [41].

The following product dimensions are evaluated [42]:∙ Pragmatic Quality (PQ): Describes the usability of

a product and indicates how successfully users are inachieving their goals using the product.

∙ Hedonic Quality Identity (HQ-I): Indicates to whatextent the product allows the users to identify with it.

∙ Hedonic Quality Stimulation (HQ-S): Mankind hasan inherent need to develop and move forward. This di-mension indicates to what extent the product can supportthis need in terms of novel, interesting, and stimulationfunctions, contents, interaction, and presentation styles.

∙ Attractiveness (ATT): Describes a global value of theproduct based on the quality perception.

Hedonic and pragmatic qualities are independent of eachother and equally contribute to the rating of UX.

AttrakDiff offers three graphs for each test. Figures 4, 5,and 6 are the portfolio graphs, in which the values of hedonicquality are represented on the vertical axis (bottom means alow value). The horizontal axis represents the value of thepragmatic quality (left means a low value). Depending on thevalues of the dimension, the tested product will lie in one ormore “character-regions”.

Figures 7, 8, and 9 feature diagrams of average valuesfor the four dimensions PQ, HQ-I, HQ-S, and ATT. WhileFigures 10, 11, and 12 show the average values for each pair

1http://attrakdiff.de/index-en.html

108

Fig. 7. In each dimension, the participants evaluated their prototypes betterthan Reddit, except in HQ-S.

Fig. 8. Facebook came out slightly better evaluated than the prototypes.

of words in the questionnaire. These two types of charts havea range that goes from −3 to 3, which represents the scaleof seven steps that each semantic differential has (higher isbetter).

C. Discussion

At first, we thought that the prototypes of participants(AUX) were always going to obtain better results in compar-ison to the tasks in the social networks (UX). However, thiswas not the case, except for one exception: Reddit.

As can be seen in Figures 4, and 7, the prototypes forsending messages were better evaluated than the real platform.We believe that this is because most of the participants hadnever used Reddit. However, this may also mean that theparticipants could not transfer their experience to an unknownplatform, since user tools did not work as they expected.

Regarding the PQ and HQ levels of confidence (lower isbest) observed in Figures 4, 5, and 6, we can say that theevaluations of social networks are more reliable than thoseof the prototypes, except in the case of Reddit. We attributedthis to the fact that the experiences were constant, i.e., whileall experimented with the same parts of the platforms, theprototypes varied more depending on each participant.

Fig. 9. Search evaluations are quite similar; YouTube has a little advantageover the prototypes.

Fig. 10. The most remarkable differences we can observe are that the par-ticipants rated their prototypes as straightforward, integrating, and pleasant.

It is interesting that the only dimension in which Redditstood out from the prototypes was in HQ-S (see Fig. 7, 8, and9). It is important to emphasise this point since it concerns allthe tests we carried out.

Although participants conceived GUIs based on their previ-ous knowledge, ideal according to their imagination, the real

109

Fig. 11. In each semantic differential, the participants evaluated similarly, butwe can observe the differences in brings me closer, presentable, and bold.

platforms offered them better experiences. We noticed thatmost of the participants made severe critics of their work,i.e., they expressed that they lacked the knowledge to createa good social network. We can easily see this perceptionin Figures 10, 11, and 12. In general, we can say that theplatforms always seemed more premium, sophisticated, andbetter developed than their prototypes, however, not by much.

The most interesting case is Reddit since it is an exampleof contrast between AUX and UX. A look at the results couldmark a possible route to improve user tools. For example, thepragmatic quality was negative (see Fig. 7), this is a big prob-lem since this tells us that users could not send their messages,or had many difficulties to do so. How to solve it? Severalclues can guide us: complicated, cumbersome, unpredictable,and discouraging are adjectives with entirely negative results(see Fig. 10). Then it would be worth rethinking if Redditasks for many steps to send a message, if the menus are in avisible and coherent position, or if the user tools are correctlylabelled.

VI. CONCLUSION AND FUTURE WORK

In this article, we present an evaluation method for the UXand AUX evaluation of user tools in social networks. Our goal

Fig. 12. YouTube and the prototypes get very close ratings, even so, we cannotice differences in professional, stylish, and captivating.

is to improve the widgets that these platforms offer, and in thisway, encourage the participation of users.

The tests carried out showed us that the users have pre-conceived ideas of how the GUIs of this type of platformsshould work, as a result of their previous experiences. Whenthey are in an unknown environment, users should be able totransfer their experience to solve new challenges. However,this transposition is complicated if the tools at their disposaldo not work familiarly, i.e., they do not behave as expected.

Offering a positive UX is a challenge faced by socialnetworks, as it is a crucial factor in their survival and growth.

As future work, we plan to analyse in more detail thedata we obtained from the tests. Similarly, we believe that itcould be better to do the three tasks for each of the testedplatforms; however, this would take more time. With thegained experience, we plan to design our own evaluation toolso that we can obtain more concrete and specialised data.

Acknowledgement. We thank CONACyT (Consejo Nacional de Ciencia y

Tecnologıa) for funding Luis Martın Sanchez Adame’s doctoral fellowship.

Scholarship number: 294598.

110

REFERENCES

[1] L.-S. Chen and P.-C. Chang, “Identifying crucial website quality factorsof virtual communities,” in Proceedings of the International MultiCon-ference of Engineers and Computer Scientists, vol. 1, 2010, pp. 17–19.

[2] C. El Morr and L. Eftychiou, Evaluation Frameworks for Health VirtualCommunities. London: Palgrave Macmillan UK, 2017, pp. 99–118.

[3] J. Preece, C. Abras, and D. Maloney-Krichmar, “Designing and evalu-ating online communities: Research speaks to emerging practice,” Int.J. Web Based Communities, vol. 1, no. 1, pp. 2–18, Jul. 2004.

[4] Y. Wang and Y. Li, “Proactive engagement of opinion leaders andorganization advocates on social networking sites,” International Journalof Strategic Communication, vol. 10, no. 2, pp. 115–132, 2016.

[5] F. S. Lee, D. Vogel, and M. Limayem, “Virtual community informatics:A review and research agenda,” JITTA: Journal of Information Technol-ogy Theory and Application, vol. 5, no. 1, p. 47, 2003.

[6] B. Shneiderman, C. Plaisant, M. Cohen, S. Jacobs, N. Elmqvist, andN. Diakopoulos, “Grand challenges for hci researchers,” interactions,vol. 23, no. 5, pp. 24–25, Aug. 2016.

[7] V. H. H. Chen and H. B. L. Duh, “Investigating user experience of onlinecommunities: The influence of community type,” in 2009 InternationalConference on Computational Science and Engineering, vol. 4, Aug2009, pp. 509–514.

[8] J. Preece, Online communities: Designing usability and supportingsocialbilty. John Wiley & Sons, Inc., 2000.

[9] L. F. Jacobsen, A. A. Tudoran, and L. Lahteenmaki, “Consumers’motivation to interact in virtual food communities - the importance ofself-presentation and learning,” Food Quality and Preference, vol. 62,pp. 8 – 16, 2017.

[10] O. Nov and C. Ye, “Why do people tag?: Motivations for photo tagging,”Commun. ACM, vol. 53, no. 7, pp. 128–131, Jul. 2010.

[11] A. Tella and B. J. Babatunde, “Determinants of continuance intentionof facebook usage among library and information science female un-dergraduates in selected nigerian universities,” International Journal ofE-Adoption (IJEA), vol. 9, no. 2, pp. 59–76, 2017.

[12] T. Zhou, “Understanding online community user participation: a socialinfluence perspective,” Internet Research, vol. 21, no. 1, pp. 67–81, 2011.

[13] J. Lamprecht, D. Siemon, and S. Robra-Bissantz, “Cooperation isn’tjust about doing the same thing – using personality for a cooperation-recommender-system in online social networks,” in Collaboration andTechnology, T. Yuizono, H. Ogata, U. Hoppe, and J. Vassileva, Eds.Cham: Springer, 2016, pp. 131–138.

[14] G. Fragidis, I. Ignatiadis, and C. Wills, “Value co-creation and customer-driven innovation in social networking systems,” in Exploring ServicesScience, J.-H. Morin, J. Ralyte, and M. Snene, Eds. Berlin, Heidelberg:Springer Berlin Heidelberg, 2010, pp. 254–258.

[15] H. T. X. Mai and S. O. Olsen, “Consumer participation in virtualcommunities: The role of personal values and personality,” Journal ofMarketing Communications, vol. 21, no. 2, pp. 144–164, 2015.

[16] T. J. McCormick, A success-oriented framework to enable co-createde-services. The George Washington University, 2010.

[17] “Ergonomics of human-system interaction - part 210: Human-centreddesign for interactive systems,” International Organization for Standard-ization, Geneva, CH, Standard [9241-210:2010(2.15)], Mar. 2010.

[18] T. Yogasara, V. Popovic, B. J. Kraal, and M. Chamorro-Koc, “Generalcharacteristics of anticipated user experience (aux) with interactiveproducts,” in Proceedings of IASDR2011: the 4th World Conference onDesign Research: Diversity and Unity. IASDR, 2011, pp. 1–11.

[19] D. Stone, C. Jarrett, M. Woodroffe, and S. Minocha, User InterfaceDesign and Evaluation, ser. Morgan Kaufmann Series in InteractiveTechnologies. San Francisco: Morgan Kaufman, 2005.

[20] J. A. Bargas-Avila and K. Hornbæk, “Old wine in new bottles or novelchallenges: A critical analysis of empirical studies of user experience,” inProceedings of the SIGCHI Conference on Human Factors in ComputingSystems, ser. CHI ’11. Vancouver, BC, Canada: ACM, 2011, pp. 2689–2698.

[21] E. Karapanos, J. Zimmerman, J. Forlizzi, and J.-B. Martens, “Userexperience over time: An initial framework,” in Proceedings of theSIGCHI Conference on Human Factors in Computing Systems, ser. CHI’09. Boston, MA, USA: ACM, 2009, pp. 729–738.

[22] V. Roto, E. Law, A. Vermeeren, and J. Hoonhout, “User experience whitepaper. bringing clarity to the concept of user experience,” in Proceedingsof Dagstuhl Seminar on Demarcating User Experience, vol. 22, Sep.2011, pp. 06–15.

[23] A. P. O. S. Vermeeren, E. L.-C. Law, V. Roto, M. Obrist, J. Hoonhout,and K. Vaananen-Vainio-Mattila, “User experience evaluation methods:Current state and development needs,” in Proceedings of the 6th NordicConference on Human-Computer Interaction: Extending Boundaries,ser. NordiCHI ’10. Reykjavik, Iceland: ACM, 2010, pp. 521–530.

[24] H. Kukka, M. Pakanen, M. Badri, and T. Ojala, “Immersive street-levelsocial media in the 3d virtual city: Anticipated user experience andconceptual development,” in Proceedings of the 2017 ACM Conferenceon Computer Supported Cooperative Work and Social Computing, ser.CSCW ’17. Portland, Oregon, USA: ACM, 2017, pp. 2422–2435.

[25] D. P. Magin, A. Maier, and S. Hess, “Measuring negative user experi-ence,” in Design, User Experience, and Usability: Users and Interac-tions, A. Marcus, Ed. Cham: Springer, 2015, pp. 95–106.

[26] G. Margetis, X. Zabulis, P. Koutlemanis, M. Antona, and C. Stephanidis,“Augmented interaction with physical books in an ambient intelligencelearning environment,” Multimedia Tools and Applications, vol. 67,no. 2, pp. 473–495, Nov 2013.

[27] C. Moser, Y. Chisik, and M. Tscheligi, “Around the world in 8workshops: Investigating anticipated player experiences of children,” inProceedings of the First ACM SIGCHI Annual Symposium on Computer-human Interaction in Play, ser. CHI PLAY ’14. Toronto, Ontario,Canada: ACM, 2014, pp. 207–216.

[28] G. Y. Sato, H. J. S. de Azevedo, and J.-P. A. Barthes, “Agent and multi-agent applications to support distributed communities of practice: a shortreview,” Autonomous Agents and Multi-Agent Systems, vol. 25, no. 1,pp. 87–129, Jul 2012.

[29] D. Wurhofer, A. Krischkowsky, M. Obrist, E. Karapanos, E. Niforatos,and M. Tscheligi, “Everyday commuting: Prediction, actual experienceand recall of anger and frustration in the car,” in Proceedings of the 7thInternational Conference on Automotive User Interfaces and InteractiveVehicular Applications, ser. AutomotiveUI ’15. Nottingham, UnitedKingdom: ACM, 2015, pp. 233–240.

[30] E. Zhang, G. Culbertson, S. Shen, and M. Jung, “Utilizing narrativegrounding to design storytelling gamesfor creative foreign languageproduction,” in Proceedings of the 2018 CHI Conference on HumanFactors in Computing Systems, ser. CHI ’18. Montreal QC, Canada:ACM, 2018, pp. 197:1–197:11.

[31] K. Peffers, T. Tuunanen, M. Rothenberger, and S. Chatterjee, “A designscience research methodology for information systems research,” J.Manage. Inf. Syst., vol. 24, no. 3, pp. 45–77, Dec. 2007.

[32] K. Laubis, M. Konstantinov, V. Simko, A. Groschel, and C. Weinhardt,“Enabling crowdsensing-based road condition monitoring service byintermediary,” Electronic Markets, Mar 2018.

[33] L. Patrıcio, N. F. de Pinho, J. G. Teixeira, and R. P. Fisk, “Service designfor value networks: Enabling value cocreation interactions in healthcare,”Service Science, vol. 10, no. 1, pp. 76–97, 2018.

[34] J. Koh, Y.-G. Kim, B. Butler, and G.-W. Bock, “Encouraging participa-tion in virtual communities,” Commun. ACM, vol. 50, no. 2, pp. 68–73,Feb. 2007.

[35] B. Apostolou, F. Belanger, and L. C. Schaupp, “Online communities:satisfaction and continued use intention.” Information Research, vol. 22,no. 4, 2017.

[36] J. Hummel and U. Lechner, “Social profiles of virtual communities,”in Proceedings of the 35th Annual Hawaii International Conference onSystem Sciences, Jan 2002, pp. 2245–2254.

[37] A. Iriberri and G. Leroy, “A life-cycle perspective on online communitysuccess,” ACM Comput. Surv., vol. 41, no. 2, pp. 11:1–11:29, Feb. 2009.

[38] J. Preece, “Sociability and usability in online communities: Determiningand measuring success,” Behaviour & Information Technology, vol. 20,no. 5, pp. 347–356, 2001.

[39] P. Morville, “Experience design unplugged,” in ACM SIGGRAPH 2005Web Program, ser. SIGGRAPH ’05. New York, NY, USA: ACM, 2005.

[40] M. Hassenzahl, M. Burmester, and F. Koller, “Attrakdiff: Ein fragebogenzur messung wahrgenommener hedonischer und pragmatischer qualitat,”in Mensch & Computer 2003. Springer, 2003, pp. 187–196.

[41] T. Walsh, J. Varsaluoma, S. Kujala, P. Nurkka, H. Petrie, and C. Power,“Axe ux: Exploring long-term user experience with iscale and at-trakdiff,” in Proceedings of the 18th International Academic MindTrekConference: Media Business, Management, Content & Services, ser.AcademicMindTrek ’14. Tampere, Finland: ACM, 2014, pp. 32–39.

[42] J. Hu, D. Le, M. Funk, F. Wang, and M. Rauterberg, “Attractivenessof an interactive public art installation,” in Distributed, Ambient, andPervasive Interactions, N. Streitz and C. Stephanidis, Eds. Berlin,Heidelberg: Springer Berlin Heidelberg, 2013, pp. 430–438.

111