towards a user engagement process model in open...

This paper was presented at The ISPIM Innovation Summit, Kuala Lumpur, Malaysia on 4-7 December 2016. The publication is available to ISPIM members at www.ispim.org.

1

Towards a User Engagement Process Model in Open Innovation

Abdolrasoul Habibipour* Luleå University of Technology Information Systems 971 87 Luleå, Sweden E-mail: [email protected]

Birgitta Bergvall-Kåreborn Luleå University of Technology Information Systems 971 87 Luleå, Sweden E-mail: [email protected] * Corresponding author

Abstract: Studies on open innovation have increasingly emphasized the role of individual users as collaborators in the innovation processes. Keeping users enthusiastically engaged throughout the information systems development (ISD) process for open innovation approaches is of crucial importance, specifically when the participation is voluntary. Although a few studies exist that have presented process models for user engagement, none of them are specifically based on the voluntary contribution of the users in an open environment, nor included the detailed attributes of the phenomenon of user drop-outs in their presented models. By combining the results of a comprehensive literature review and a qualitative case study of user engagement, this study aims to develop a user engagement process model that includes the variety of reasons for user drop-out, and then to present some practical guidelines that would facilitate prolonged user engagement in the ISD process in an open innovation environment.

Keywords: Open Innovation; User Engagement; User Involvement; Voluntary Contribution; User Drop-out; Process Model; Information Systems Development.

1 Introduction

Open innovation has become a key paradigm in innovation management (Chesbrough 2006; West et al. 2014) and a very popular research topic in management and information systems (IS) disciplines (Gassmann et al. 2010). One of the more common definitions of open innovation states that "firms can and should use external ideas as well as internal ideas, and internal and external paths to market, as firms look to advance their technology" (Chesbrough 2006, p. 1). Following this definition, external knowledge sourcing becomes a key function in open innovation (Bengtsson et al. 2015; Ghisetti et al. 2015; Ooms et al. 2015; Rogbeer et al. 2014), and individual users as contributors in the innovation process have been seen as a valuable external resource (Jespersen 2010). Despite this, the

http://www.ispim.org/

mailto:[email protected]

mailto:[email protected]


2

perspective of contributors is seldom included; instead they are treated merely as an effective and efficient way to improve innovation processes (see, for example, Gassmann et al. 2010; Lilien et al. 2002; West et al. 2014).

Within the field of information systems (IS), on the other hand, there is a long research tradition related to participatory design and user involvement (Bansler 1989; Ehn and Kyng 1987; Iivari and Lyytinen 1998). This research is strongly grounded in democratisation and empowerment values and highlights the perspective of the users (Bergvall-Kareborn et al. 2009; Ståhlbröst and Bergvall-Kåreborn 2013). While the users within IS research tend to be end-users linked to an organization, open innovation instead focuses on voluntary contributors without organizational ties that may not necessarily be end-users or customers of the prototyped system or service (Ståhlbröst and Bergvall-Kåreborn 2013).

Common to both fields, open innovation and information systems, is their focus on users as active participants and sources of knowledge and experience (Hartwick and Barki 1994; Jespersen 2010; Vines et al. 2013). Thus, involving individual users in the process of systems development is a key dimension of both open innovation and information systems that contributes positively to new innovations as well as system success, system acceptance and user satisfaction (Bano and Zowghi 2015; Leonardi et al. 2014; Lin and Shao 2000). However, creating a sustainable user engagement is not an easy task as the attention, motivations and expectations of users and contributors tend to decrease over time making it difficult to maintain the same level of engagement throughout the lifetime of a project (Carr 2006; Ley et al. 2015; Ogonowski et al. 2013). Previous studies also show that users tend to drop-out before the activity or project they signed up for is completed (Habibipour et al. 2016). This drop-out is not linked to specific phases but occurs throughout the development process, from contextualization to test and evaluation (Habibipour et al. 2016). The importance of keeping users motivated during the whole participation period is acknowledged in the IS literature, along with costs associated with drop-out - decreased time and cost efficiency, reduced quality assurance, the loss in established mutual trust and users' accumulated understanding about the project or an activity - (Ley et al. 2015; Sambamurthy and Kirsch 2000).

What is missing are guidelines, process models and theories describing what we should do and how we should act in order to keep users motivated and reduce the likelihood of user drop-out throughout the innovation and development process. The phenomenon of user drop-out is particularly challenging in contexts where the participation is voluntary and there is no organizational leverage that can be used to secure participation (Ståhlbröst and Bergvall-Kåreborn 2013). This includes many of the more recent development and innovation approaches, such as user-driven innovation (De Moor et al. 2010), crowdsourcing (Howe 2006), and living Lab (Bergvall-Kareborn et al. 2009; Ståhlbröst 2008). Therefore, understanding the phenomenon of user-drop-out and how to motivate users to engage and stay engaged through an activity or project is of crucial importance for open innovation approaches.

Hence, the aim of this paper is to develop a user engagement process model that includes the variety of reasons for user drop-out and how this can be managed. To achieve this goal, we have used the results of a literature review carried out by the authors as well as a qualitative case study to identify key factors influencing user drop-out behavior within innovation and development processes and how the user engagement process can be designed to reduce the likelihood of user drop-out. Based on the proposed model, we present practical guidelines that would facilitate prolonged user engagement in the ISD process in an open innovation environment. The paper ends with some concluding remarks.



3

2 Background

Opening up the innovation process by involving users in different innovation activities is a key factor in inbound open innovation (Chesbrough and Crowther 2006). Involving users in the information systems development (ISD) process is of crucial importance (He and King 2008; Sambamurthy and Kirsch 2000) and have been so since the 1960’s when the participatory design tradition was introduced (Bansler 1989; Ehn and Kyng 1987; Iivari and Lyytinen 1998). Previous studies within IS literature also clarify important differences between what might first seem like synonymous concepts such as participation, involvement, and engagement. For example, participation is characterized by the actual behaviour of users, such as carrying out certain activities; involvement represents a subjective psychological state where users reflect on the importance and personal relevance of a system (Barki and Hartwick 1989; Barki and Hartwick 1994; Hartwick and Barki 1994); and influence refers to the users’ actual power position within the development team and requires that users affect the outcomes of the design process (Baroudi et al. 1986). Finally, the term user engagement is a more general term that in many ways includes both participation and involvement (Bano and Zowghi 2015) by emphasising active participation or involvement of users throughout the ISD process (O'Brien and Toms 2008).

Within the IS literature, the term “process” has acquired several meanings and definitions varying from causal explanation, a category of concepts or a sequence of events (Van de Ven 1992). When the process is seen as causal explanation, the relationship between independent and dependent variables in the process is more highlighted. In this viewpoint, events and activities within the process have not been explicitly examined. The process can be considered as a category of concepts when the focus is explicitly on the actions of individuals or organizations within the process. When it comes to a sequence of events, the process describes how things change over time. The results of a comprehensive literature review conducted by Sambamurthy and Kirsch (2000) indicate that the role of users and particularly the concept of “user engagement” are more highlighted in the ISD literature when the process is seen as a category of concepts. From this point of view, it is very important to understand how the process concepts are related to each other and how these core concepts influence the process of system development. They also identified tasks (i.e., work done to construct an information system) as another core concept in this process which might include planning, feasibility study, analysis, design, coding, testing, implementation, and maintenance. Pekkola et al. (2006) argue that IS research still lacks insights on how the user engagement should take place in different phases of the ISD process. In order to bridge the gap between participatory design and ISD process, they proposed an iterative ISD process comparing and combining ISD process with the key elements of participatory design and suggested to gain every participant committed in the various stages of the ISD process.

Considering user engagement as an “ongoing process”, O’Brien and Toms (2008) developed a model for user engagement based on an extensive literature review in the context of human–computer interaction. They presented an iterative process model for user engagement consists of four core components namely, point of engagement, period of engagement, disengagement, and re-engagement. Each stage, in turn, represents different internal and external attributes. However, the relationship between some of its attributes in the various stages is not clearly stated. In their process model, the point of engagement is initiated by users' motivations and interests. Engagement is sustained by factors like



4

awareness, interactivity, novelty, reciprocal feedback and interest. User disengagement1 might occur for various reasons such as interruption, usability problems and challenges. When it comes to re-engagement, the attributes of this component are the same attributes of the point of engagement. In this study, using the term re-engagement raises an important question of when a user can be considered as a dropped out or disengaged user. Although very few studies in either the ISD or open innovation literature have offered a definition of the concept of user drop-out, the possible explanation for this phenomenon is “when participants made an internal decision to stop the activity or when factors in the participants’ external environment caused them to cease being engaged” (O'Brien and Toms 2008, p. 944). Accordingly, we apply this definition in our study because it includes both internal and external influential factors on user drop-out behavior.

There are also attempts to present a process model for user engagement that are limited to one specific activity. One such example is Georges et al.’s (2015) proposed user engagement model for field trials, aiming to explain the factors that affect the engagement of end-users to test innovations in real-life environments. In this model, however they include some possible influential factors on user drop-out behavior in general terms such as perceived usefulness, perceived ease of use, uncertainty and functional maturity, but these terms have not been discussed in greater details and different attributes of each concept of the presented model.

Although the above mentioned studies (Georges et al. 2015; O'Brien and Toms 2008; Pekkola et al. 2006) have presented process models for user engagement, none of them are specifically based on the voluntary contribution of the users in an open environment, nor included the detailed attributes of the phenomenon of user drop-outs in their presented models.

3 Research Method

In this study, a "user engagement process model" for open innovation is presented. We ended up using the term “user engagement” since we are interested in the broad spectrum of including users (Bano and Zowghi 2015). Moreover, “user engagement” put more emphasis on active participation or involvement of users throughout the ISD process (O'Brien and Toms 2008). The model is grounded on the results of a previously conducted literature review (Habibipour et al. 2016) as well as a qualitative case study. This triangulation of the data strengthens the validity of the presented model and makes our results more reliable (Benbasat et al. 1987).

The literature review followed a structured concept-centric approach based on Webster and Watson’s recommendations (2002) and aimed to identify, categorize and sum up existing research on why people drop-out of user studies before the activity or project they signed up for ended. Due to the interdisciplinary and emerging nature of the area, we used four sets of search terms and developed a logic for combing them, instead of using a given set of keywords or predefined publication sources. For a more detailed description of the methodology and the study as a whole, see Habibipour et al. (2016).

The case study followed an exploratory research method as outlined by Yin (1994). We employed this research method because it is more suitable for preliminary studies that the boundaries between the context and phenomena are not yet clear. Moreover, this approach 1 In this work, the authors used the term disengagement synonymously with drop-out.



5

is more appropriate to explore and examine “why” and “how” questions (Yin 1994). We involved 118 users in the development and test of a 'privacy enhancing application' carried out in an open innovation environment. The aim of the study was to engage users to test the application and suggest how it could be further improved while at the same time collect data on user drop-out. The duration of the data collection was 34 days and included three data sources: 1) direct observation of users and their behaviour during the test phase; 2) email communication with users during the test phase regarding information about how to carry out the test, technical problems that occurred during the test, and other problems they experienced; and 3) an online semi-structured questionnaire carried out immediately after the user study had ended. The questionnaire was designed based on the results from the above mentioned literature review (Habibipour et al. 2016), and focused on actual reasons for the users drop-out. It was sent to users who had signed up for the study but not fulfilled the tasks assigned to them, i.e. dropped out from the study. One reminder was sent to all users who did not complete the assigned tasks within the scheduled time and if we did not hear from them within 3-4 days after that reminder, we considered them as “dropped out users”. This resulted in 91 user drop-outs.

Combining and comparing the results of the literature review and the case study, a preliminary "user engagement process model" was developed including a variety of reasons for user drop-out throughout the ISD process. By focusing on the core concepts and detailed attributes of the developed process model, a total of 14 step-by-step practical guidelines are proposed to minimize the likelihood of user drop-out. A summary of the research study is shown in Figure 1.

Figure 1 Research process

4 Results

In this section, we present the findings of our study. We first present and summarize the results of the previously conducted literature review (Habibipour et al. 2016) and then, report our findings from the case study.

4.1 State-of-the-Art from a Literature Review The literature review focused on reasons for user drop-out, (Habibipour et al. 2016) and resulted in 44 articles. The notable point is that most of the reviewed articles (29 of 44) were associated with emerging IS topics such as crowdsourcing (9 articles), crowd-sensing (7 articles) and Living Labs (13 articles), which are mainly based on voluntary contribution in an open innovation environment. Only 12 studies focused on the traditional approaches of user engagement such as user-centered design (6 articles) and participatory design (6 articles), which mainly have an organizational focus.

This study illustrates that there are various reasons that might lead to user drop-out. The findings also confirm that user drop-out is likely to happen due to different internal and external influential factors in different phases of the ISD and user engagement process. The



6

authors of the study identified more than 30 influential factors on user drop-out behavior that might be associated with task design, scheduling, user selection process, user preparation, implementation and test process, and interaction with the users. Based on this classification, we summarize the findings of the literature review in Table 1.

Table 1 Summary of findings from the literature review (number of articles: 44)

Drop-out reason Number of hits

Task Design Task complexity (hardness of use) 7

Usability problems (instability or unreliability) 9

High task-dependency or task-autonomy 3

Limitation of users' resources 5 Scheduling and timing Longevity (or long gaps between phases) 3

Inappropriate season of year 2

Limitation of users time 7 User selection process Wrong user selection 5

High number of users 4 User preparation Unclear guideline (lack of documentation) 3

Inadequate sensitization or training 3

Implementation and test process Inadequate infrastructure 3

Insufficient technical support 3

Privacy and security concerns 8

Inappropriate monetary reward 6

Unrealistic users expectation or routine 5

Disappearing novelty aspects 2

Interaction with the users Issues related with contact person 11

Lack of mutual trust 12

Insufficient feedback to users 11

Ignore users’ feedback, idea or need 5

Lack of interrelationship between participants 5

4.2 The Case Study of User Engagement in the ISD Process When it comes to the case study, a total of 118 users showed an interest to participate in the user study and filled out the initial survey. Of these, only 27 users completed all assigned tasks while 91 users (77%) dropped out. Out of the 91 questionnaires that we sent out to all users who dropped out, only 32 of them completed the drop-out questionnaire. An in-depth



7

analysis was conducted based on the collected data (our direct observation, email communication and the drop-out questionnaire).

The results of our data analysis showed that instability and non-functionality of the prototype was the most influential factor on the users’ decision to drop-out of the project. Some users were also dissatisfied with the test due to the complexity of the tasks, too long and exhausting tasks, lack of clear instructions on how to do the tasks, and incompatibility with the smartphones. Insufficient reminders and too strict deadlines were other influential factors, as well as, the timing of the test and lack of time. Some users dropped out due to limited access to a computer or internet connection. Finally, privacy and security concerns were also identified as important barriers for the users to stay engaged throughout the user study.

5 Proposed Process Model for User Engagement

In this section, we present a user engagement process model for open innovation approaches that includes the variety of reasons for user drop-out. In order to develop this model, we use the results of the literature review and case study as input for our analysis on the rationale behind the identified influential factors on user drop-out behavior. The reason behind this is that, data from two or more sources offers the opportunity for triangulation and increases the reliability of the results and greater support to the researcher's conclusions (Benbasat et al. 1987; Yin 1994). Interestingly, most of the identified reasons for user drop-out in the case study were in accordance with the findings of the literature review. However, the case study revealed other reasons that have not been discussed in the literature review such as the importance of reminders and the influence of close and strict deadlines on user drop-out behavior. On the other hand, some of the drop-out reasons that have been mentioned in the literature review were not applicable in the case study due to nature of the user study. For example, the user study was relatively a short term activity in an open environment and it was not possible to investigate the influence of social activities and interrelationship between participants.

By combining the results of the literature review and the case study, we can confirm that the influential factors on user drop-out behaviour are associated with six main concepts, namely: 1) task design, 2) scheduling, 3) user selection process, 4) user preparation, 5) implementation and test process, and 6) interaction with the users. Inspired by seeing the process as a category of concepts (Sambamurthy and Kirsch 2000; Van de Ven 1992), we consider these six tasks as the core concepts of the developed model. Each of these core concepts, in turn, represents different influential factors on user drop-out behavior. These factors might have either positive (+) or negative (-) influence on sustainable user engagement. Figure 2 shows our presented “user engagement process model”.

In the following, we discuss each of these core concepts and their attributes and propose a total of 14 practical guidelines that will help the organizers of a user study to minimize the likelihood of user drop-out through the ISD process in an open innovation environment. However, we cannot yet generalize our recommendations to all possible cases of user engagement since our focus in this study is on voluntary contribution of the participants and the nature of involving users within the organizations in the IS context is inherently different with the voluntary contribution in an open innovation environment. In addition, the expectations and motivations of voluntary contributors may also be different compared to paid-employees in the organization (Ståhlbröst and Bergvall-Kåreborn 2013). On the



8

other hand, our findings also confirm that user drop-out behavior is influenced by other affective factors such as degree of openness, methodology of ISD, level of user engagement, motivation type and activity type and thus, some of these guidelines may not be applicable in all situations.

Figure 2 Proposed “User Engagement Process Model”

5.1 Task Design Regarding the task design, our model suggests that “the task should be designed as easy to use, easy to understand and autonomous as possible and if it is not possible to simplify the task enough, it should be divided to sub-tasks” (Guideline 1). Based on this guideline, it is recommended that the tasks should be clearly defined, be explicitly specified and be highly autonomous with a fixed deadline per task. However, Kobren et al. (2015) argue that setting specific goals in the long-term projects may be disadvantageous for the user engagement, as users immediately tend to drop-out upon finishing a task. Moreover, Ley et al. (2015) in a comparison between two different long-term projects mentioned that it is not totally clear for us whether a more explicit definition of project (e.g., roles, method, etc.) at the beginning of the project would have positive or negative effect on the user retention. Therefore Guideline 1 needs to be evaluated and investigated further in different types of user studies.

As mentioned, most of the drop-outs in the case study occurred due to instability or non-functionality of the prototype. Even minor usability problems could discourage participants and users must be informed that the prototypes are not as stable and reliable as commercial technologies. Thus we state: “high functionality and stability of the prototype enhance the reliability of the system and increases the likelihood of user retention” (Guideline 2). However, Guideline 2 might have different effects in different phases of ISD process. For instance, in the test phase, a highly mature prototype can demotivate the users as their ability to provide a proper feedback decreases (Georges et al. 2015). Hence, the level of functional maturity in the test phase needs to be considered as an exception and should be investigated further.



9

Limitation of users’ resources (such as mobile battery or mobile data quota) is another important factor influencing users’ willingness to retain in the user study. Thus, “the task should be designed in a way to avoid unnecessary data transmission with the user’s device as far as possible” (Guideline 3). A possible solution for this issue is that the organizers of a user study provide the users with necessary resources, tools and equipment.

5.2 Scheduling The results of the literature review as well as the case study confirm that users might give up participating in the activity due to different issues related to the scheduling. Combining the results, we propose: “long gaps between the project’s phases, longevity of the activity and inappropriate time of the year to involve users in the user study are likely to have negative effects on users' motivation and should be avoided. Moreover, in order to reduce the likelihood of users’ forgetfulness, reminders must be set in the schedule of the ISD” (Guideline 4). Regarding the Guideline 4, the rapid ISD methodologies such as agile development might have burdened the users less than the other older ISD methodologies (e.g., waterfall or prototyping) by requiring shorter periods of user engagement (Hanssen and Fægri 2006; Hoda et al. 2010). Thus, it is plausible that a methodology implying a smaller user effort is associated with smaller user drop-outs. However, in such rapid methodologies, applying an optimal and precise scheduling is mostly important as the study period is not usually flexible, and users are not able to involve in the project at their own pace and this issue can also discourage users because it needs their availability.

5.3 User Selection Process Finding motivated and long-term engaged users is not an easy task. As Ståhlbröst and Bergvall-Kåreborn (2013) emphasized, the users that are involved in the projects should be not only contributors, but also qualified contributors. In many cases users are selected based on their visible characteristics because these specifications are more easily identifiable. But, their invisible characteristics would be more important in the tasks that need to be more creative and innovative (Bergvall-Kåreborn and Larsson 2008). “In order to identify and select right users, a reasonable balance must be struck between their visible characteristics such as age, gender and education on the one hand and their invisible traits such as personal value and technical skills on the other hand” (Guideline 5). However, the criteria for selecting participants for the projects are different and these criteria may vary according to the project nature (i.e., long-term or short-term user studies, crowdsourcing activities, Living Lab oriented development, user innovation projects, etc.). Therefore, Guideline 5 may have to be expanded to include and address these issues depending on the nature, type, phase and duration of the user study. It is worth noting that, however identifying potential end-users of an Information system is not an easy task as the final service or product doesn’t exist yet, but it would be beneficial and useful to include them within the ISD process (Ståhlbröst and Bergvall-Kåreborn 2013).

Besides the importance of identifying right participants, previous literature show that by increasing the number of users, the average level of participation falls down (Habibipour et al. 2016). “The smaller number of participants reduces the likelihood of user drop-out since it will be easier to interact with them by maximize reciprocal feedback between the organizers and users” (Guideline 6). Although Guideline 6 recommends the smaller number of users, Hess et al.’s (2008) argue that by involving a larger group of people in a project, the result is more reasonable and reliable. Moreover, especially in open innovation processes, the number of ideas and feedbacks decreases in a smaller group of participants



10

(Ståhlbröst and Bergvall-Kåreborn 2013). Therefore, it is very important to define a minimum number of participants which can be filled up with new users (Hess et al. 2008).

5.4 User preparation Regarding the user preparation, our results confirm that a clear guideline would minimize the possibility of users’ confusion and their discouragement. During the case study, some users argued that they dropped out due to inaccessibility or ambiguity of provided guidelines or instructions on how to perform the tasks. “The organizers must make the users aware and well-informed about the whole process of engagement by providing them a clear, accessible and comprehensible guideline before and during the participation process” (Guideline 7).

5.5 Implementation and Test Process Within the ISD process, the provided infrastructure should be adequate in a way that enables technologies at various stages of development (Ley et al. 2015). Thus, “besides a timely and appropriate technical support, building an adequate infrastructure by providing a flexible test plan and easily available and accessible prototype can increase the likelihood of long-term and stable user engagement” (Guideline 8). Moreover, users may also be discouraged from participating after the novelty aspects of the prototype disappeared, and by continuous advancement of the markets in the area of the study, they would prefer to not use the system any longer (Ley et al. 2015). Therefore, “the developers should try to keep the prototype updated as far as possible. As the prototype becomes outdated over time, it is difficult to keep the users enthusiastically engaged in the longitudinal user study” (Guideline 9).

Privacy and security issues are also influential factors on user drop-out behavior and have been paid much attention in the previous literature (Padyab 2014; Ståhlbröst et al. 2015). Moreover, the results of the case study confirm that “privacy protection is positively associated with sustainable user engagement. Allowing participants to delete their personal data can increase the likelihood of long-term and stable user engagement” (Guideline 10).

Several reviewed studies discussed incentivizing users by appropriate economic rewards and emphasize this point as an influencing factor on user’s motivation. “The amount of the received monetary reward should be proportional to the time users spent on their participation period. Moreover, incentivize users by providing periodical micro payments within their participation period can increase the likelihood of long-term user engagement” (Guideline 11). However, in the creative and innovative activities, monetary reward is considered as less influential than the intrinsic motivators (Ståhlbröst and Bergvall-Kåreborn 2013). Accordingly, by considering this fact that extrinsic and intrinsic motivational factors can influence each other or even may have some conflicts, further research is needed in order to better understand how best to mix them and how that varies with project’s longevity and type.

5.6 Interaction with the Users Prior studies have shown that if a participant is contacted several times by different persons, the likelihood of dropping out will be higher (Habibipour et al. 2016). “Contact point should be as clear and easy to access as possible. A single point of contact facilitates the mutual trust and will increase the likelihood of user retention” (Guideline 12).



11

Regarding the interaction with the users, it is necessary to give the users this feeling that their contribution is valuable. Therefore, “all users’ feedback must be taken into account and not be ignored. The users should also be informed about the influence of their participation” (Guideline 13). In some cases, a user (e.g., lead participant) can play the role of a promoter inside the group in order to increase the level of mutual trust and also may facilitate and mediate reciprocal feedback.

Besides the relationship between participants and developers, the relationship between participants themselves is also important. “An interrelationship between participants by providing a social space and use of social media is likely to have positive effects on users' motivation and reduces the likelihood of user drop-out” (Guideline 14). In this way, online social networks may facilitate the interrelationship between participants. However, organizers of a user study should keep it in mind that participants’ relationship could have some negative effects because unhappy participant will tell other participants about their dissatisfaction and it can demotivate other users to complete the task (Carr 2006).

6 Concluding Remarks

In this study, we developed a user engagement process model that includes the variety of reasons for user drop-out in an open innovation environment. The core concepts of the presented model were extracted from the results of a comprehensive literature review on the documented reasons for user drop-out as well as the results of a qualitative case study in an open innovation environment. By focusing on all steps of user study from design tasks to interaction with the users, such a model can enrich the innovation management literature by enabling us to build a sustainable user engagement in an open innovation environment. More importantly, a comprehensive understanding of how users in an open environment should be involved in the ISD process requires step-by-step practical guidelines to design and conduct a user study. By providing a user engagement process model and practical guidelines to design and conduct a user study in an open innovation environment, organizers of a user study will be protected from the negative consequences associated with user drop-out such as extra costs and time delays. While benefiting from open innovation, these guidelines all have been presented in order to facilitate the process of user engagement and reduce the likelihood of user drop-out and should not be lead us to take control of the participants and restrict them.

This study also opens up several avenues for future research. As mentioned, O’Brien and Toms (2008) introduce re-engagement as one of the core concepts of their user engagement process model. An interesting topic for further research would be to clarify how and why user motivation for engaging and staying engaged differ. And more specifically, how the organizers of a user study can re-motivate the dropped out users in order to re-engage them and what are the benefits of doing so. As a final remark, the presented guidelines are still to be debated and might be expanded depending of the nature of the user study such as degree of openness, methodology of ISD, level of user engagement, motivation type and activity type. We believe that far more work should be done to investigate and evaluate each of the presented guidelines and it creates numerous opportunities for the future research in this area.



12

References

Bano, M. & Zowghi, D. 2015, "A systematic review on the relationship between user involvement and system success", Information and Software Technology, vol. 58, pp. 148-169.

Bansler, J. 1989, "Systems development research in Scandinavia: three theoretical schools", Scandinavian journal of information systems, vol. 1, no. 1, pp. 3-20.

Barki, H. & Hartwick, J. 1994, "Measuring user participation, user involvement, and user attitude", MIS Quarterly, vol. 18, no. 1, pp. 59-82.

Barki, H. & Hartwick, J. 1989, "Rethinking the concept of user involvement", MIS Quarterly, vol. 13, no. 1, pp. 53-63.

Baroudi, J.J., Olson, M.H. & Ives, B. 1986, "An empirical study of the impact of user involvement on system usage and information satisfaction", Communications of the ACM, vol. 29, no. 3, pp. 232-238.

Benbasat, I., Goldstein, D.K. & Mead, M. 1987, "The case research strategy in studies of information systems", MIS Quarterly, vol. 11, pp. 369-386.

Bengtsson, L., Lakemond, N., Lazzarotti, V., Manzini, R., Pellegrini, L. & Tell, F. 2015, "Open to a select few? Matching partners and knowledge content for open innovation performance", Creativity and innovation management, vol. 24, no. 1, pp. 72-86.

Bergvall-Kareborn, B., Holst, M. & Stahlbrost, A. 2009, "Concept design with a living lab approach", In Proceedings of the 42nd Hawaii International Conference on System Sciences, IEEE (2009), pp. 1-10.

Bergvall-Kåreborn, B. & Larsson, S. 2008, "A case study of real-world testing", Proceedings of the 7th International Conference on Mobile and Ubiquitous Multimedia, ACM (2008), pp. 113-116.

Carr, C.L. 2006, "Reciprocity: The golden rule of IS-user service relationship quality and cooperation", Communications of the ACM, vol. 49, no. 6, pp. 77-83.

Chesbrough, H. 2006, "Open innovation: a new paradigm for understanding industrial innovation", In Open innovation: Researching a new paradigm, Oxford: Oxford University Press, pp. 1-12.

Chesbrough, H. & Crowther, A.K. 2006, "Beyond high tech: early adopters of open innovation in other industries", R&d Management, vol. 36, no. 3, pp. 229-236.

De Moor, K., Berte, K., De Marez, L., Joseph, W., Deryckere, T. & Martens, L. 2010, "User-driven innovation? Challenges of user involvement in future technology analysis", Science and Public Policy, vol. 37, no. 1, pp. 51-61.

Ehn, P. & Kyng, M. 1987, "The collective resource approach to systems design", Computers and democracy-A Scandinavian challenge. Avebury, Aldershot, England, 1987.

Gassmann, O., Enkel, E. & Chesbrough, H. 2010, "The future of open innovation", R&d Management, vol. 40, no. 3, pp. 213-221.

Georges, A., Schuurman, D., Baccarne, B. & Coorevits, L. 2015, "User engagement in living lab field trials", Info, vol. 17, no. 4, pp. 26-39.



13

Ghisetti, C., Marzucchi, A. & Montresor, S. 2015, "The open eco-innovation mode. An empirical investigation of eleven European countries", Research Policy, vol. 44, no. 5, pp. 1080-1093.

Habibipour, A., Bergvall-Kareborn, B. & Ståhlbröst, A. 2016, "How to Sustain User Engagement over Time: A Research Agenda", Proceedings of Twenty-second Americas Conference on Information Systems, San Diego, 2016.

Hanssen, G.K. & Fægri, T.E. 2006, "Agile customer engagement: a longitudinal qualitative case study", Proceedings of the 2006 ACM/IEEE international symposium on Empirical software engineering, ACM, , New York, NY, USA, 2006 pp. 164-173.

Hartwick, J. & Barki, H. 1994, "Explaining the role of user participation in information system use", Management science, vol. 40, no. 4, pp. 440-465.

He, J. & King, W.R. 2008, "The role of user participation in information systems development: implications from a meta-analysis", Journal of Management Information Systems, vol. 25, no. 1, pp. 301-331.

Hess, J., Offenberg, S. & Pipek, V. 2008, "Community driven development as participation?: involving user communities in a software design process", Proceedings of the Tenth Anniversary Conference on Participatory Design, 2008, Indiana University, pp. 31-40.

Hoda, R., Noble, J. & Marshall, S. 2010, "Agile undercover: when customers don’t collaborate", Agile Processes in Software Engineering and Extreme Programming, Springer, pp. 73-87.

Howe, J. 2006, "The rise of crowdsourcing", Wired magazine, vol. 14, no. 6, pp. 1-4.

Iivari, J. & Lyytinen, K. 1998, "Research on information systems development in Scandinavia—unity in plurality", Scandinavian Journal of Information Systems, vol. 10, no. 1-2, pp. 135-185.

Jespersen, K.R. 2010, "User-involvement and open innovation: The case of decision-maker openness", International Journal of Innovation Management, vol. 14, no. 03, pp. 471-489.

Kobren, A., Tan, C.H., Ipeirotis, P. & Gabrilovich, E. 2015, "Getting more for less: optimized crowdsourcing with dynamic tasks and goals", Proceedings of the 24th International Conference on World Wide Web International World Wide Web Conferences Steering Committee, pp. 592-602.

Leonardi, C., Doppio, N., Lepri, B., Zancanaro, M., Caraviello, M. & Pianesi, F. 2014, "Exploring long-term participation within a living lab: satisfaction, motivations and expectations", Proceedings of the 8th Nordic Conference on Human-Computer Interaction: Fun, Fast, Foundational, ACM, pp. 927-930.

Ley, B., Ogonowski, C., Mu, M., Hess, J., Race, N., Randall, D., Rouncefield, M. & Wulf, V. 2015, "At home with users: a comparative view of Living Labs", Interacting with Computers, vol. 27, no. 1, pp. 21-35.

Lilien, G.L., Morrison, P.D., Searls, K., Sonnack, M. & Hippel, E.v. 2002, "Performance assessment of the lead user idea-generation process for new product development", Management science, vol. 48, no. 8, pp. 1042-1059.



14

Lin, W.T. & Shao, B.B. 2000, "The relationship between user participation and system success: a simultaneous contingency approach", Information & Management, vol. 37, no. 6, pp. 283-295.

O'Brien, H.L. & Toms, E.G. 2008, "What is user engagement? A conceptual framework for defining user engagement with technology", Journal of the American Society for Information Science and Technology, vol. 59, no. 6, pp. 938-955.

Ogonowski, C., Ley, B., Hess, J., Wan, L. & Wulf, V. 2013, "Designing for the living room: long-term user involvement in a living lab", Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ACM, pp. 1539-1548.

Ooms, W., Bell, J. & Kok, R.A. 2015, "Use of social media in inbound open innovation: Building capabilities for absorptive capacity", Creativity and Innovation Management, vol. 24, no. 1, pp. 136-150.

Padyab, A.M. 2014, "Getting More Explicit On Genres Of Disclosure: Towards Better Understanding Of Privacy In Digital Age (Research In Progress)", Norsk konferanse for organisasjoners bruk av IT, 2014.

Pekkola, S., Kaarilahti, N. & Pohjola, P. 2006, "Towards formalised end-user participation in information systems development process: bridging the gap between participatory design and ISD methodologies", Proceedings of the ninth conference on Participatory design: Expanding boundaries in design-Volume 1, ACM, pp. 21-30.

Rogbeer, S., Almahendra, R. & Ambos, B. 2014, "Open-Innovation Effectiveness: When does the Macro Design of Alliance Portfolios Matter?", Journal of International Management, vol. 20, no. 4, pp. 464-477.

Sambamurthy, V. & Kirsch, L.J. 2000, "An integrative framework of the information systems development process", Decision Sciences, vol. 31, no. 2, pp. 391-411.

Ståhlbröst, A. 2008, "Forming future IT - The living lab way of user involvement", Doctoral dissertation, Luleå tekniska universitet, 2008.

Ståhlbröst, A. & Bergvall-Kåreborn, B. 2013, "Voluntary contributors in open innovation processes" in Managing Open Innovation Technologies Springer, pp. 133-149.

Ståhlbröst, A., Padyab, A., Sällström, A. & Hollosi, D. 2015, "Design Of Smart City Systems From A Privacy Perspective", IADIS International Journal on WWW/Internet, vol. 13, no. 1, pp. 1-16.

Van de Ven, A. 1992, "Suggestions for studying strategy process: A research note", Strategic Management Journal, vol. 13, no. 5, pp. 169-188.

Vines, J., Clarke, R., Wright, P., McCarthy, J. & Olivier, P. 2013, "Configuring participation: on how we involve people in design", Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ACM, pp. 429-438.

Webster, J. & Watson, R.T. 2002, "Analyzing the past to prepare for the future: Writing a literature review", MIS Quarterly, vol. 26, no. 2, pp. 13-23.

West, J., Salter, A., Vanhaverbeke, W. & Chesbrough, H. 2014, "Open innovation: The next decade", Research Policy, vol. 43, no. 5, pp. 805-811.

Yin, R. 1994, Case study research: Design and methods. Second edn, Thousand Oaks, Sage publishing, Beverly Hills.


towards a user engagement process model in open...

Documents