IN DEGREE PROJECT COMPUTER SCIENCE AND ENGINEERING,SECOND CYCLE, 30 CREDITS

, STOCKHOLM SWEDEN 2019

How In-House Usability Testing Increases Product Development

JONATAN SEGER

KTH ROYAL INSTITUTE OF TECHNOLOGYSCHOOL OF ELECTRICAL ENGINEERING AND COMPUTER SCIENCE

ABSTRACT Can the Product Development Process at Tcom be improved by moving usability testing in-house?

The interest in this question lies in the agile transformation Tcom initiated in early 2018. Once agile,

the company’s interests are efficiency and increasing quality. One way of increasing the quality of

their digital products is to improve the process behind it, where involving users in usability testing

should be central in an iterative development process. A qualitatively driven mixed methods research

approach facilitated the acquisition of the empirical data necessary to successfully answer the research

question. The culture of Tcom’s usability testing was collected via a survey. The survey was

complemented with the current testing process through four interviews and two observations of

usability tests, one internal and one external. Parameters between the two tests were compared and

weighted against each other. The main findings reveal that the internal process shows promising

characteristics. It is, up to, 1/10 as expensive per participant, has a quicker call-to-test-time, facilitates

transparency, and allows refinement of the process. With the main findings in hand, the conclusion

points to the fact that usability testing should be composed and conducted by Tcom itself. The greatest

challenge to such a change is how a usability perspective can be integrated into the Scrum

methodology.

SAMMANFATTNING Kan produktutvecklingsprocessen på Tcom förbättras genom att flytta användbarhetstester internt?

Intresset i denna fråga ligger i den agila transformation som företaget påbörjade tidigt 2018.

Effektivitet och kvalitet är centrala förbättringspunkter med det agila arbetssättet. Ett sätt att öka

kvaliteten på deras digitala produkter är att förbättra processen bakom, där involveringen av användare

i användbarhetstester bör vara centralt i ett iterativt arbetsförlopp. En kvalitativt driven mixed methods

research metod tillämpades för att förvärva den empiriska data som krävdes för att framgångsrikt

besvara frågeställningen. Tcoms användbarhetstestkultur undersöktes via en enkät. Denna metod

kompletterades med 4 intervjuer och 2 observationer av användbarhetstest, ett externt och ett internt.

Parametrar från de två observationerna ställdes mot varandra. De centrala upptäckterna avslöjar att den

interna processen uppvisar lovande egenskaper. Den är upp till en tiondel så kostsam per deltagare, har

snabbare beställning-till-test-tid, tillåter transparens, och möjliggör förbättring av själva testprocessen.

Med hänsyn till huvudfynden kan vi konkludera att användbarhetstest bör utformas och utföras av

Tcom själva. Den största utmaningen till en sådan förändring ligger i hur ett sådant

användbarhetsperspektiv ska implementeras i det agila ramverket Scrum.

How In-House Usability Testing Increases Product Development

Jonatan Seger

KTH, Royal Institute of Technology

Stockholm, Sweden

jseger@kth.se

ABSTRACT

Can the Product Development Process at Tcom be

improved by moving usability testing in-house? The

interest in this question lies in the agile transformation

Tcom initiated in early 2018. Once agile, the company’s

interests are efficiency and increasing quality. One way of

increasing the quality of their digital products is to improve

the process behind it, where involving users in usability

testing should be central in an iterative development

process. A qualitatively driven mixed methods research

approach facilitated the acquisition of the empirical data

necessary to successfully answer the research question. The

culture of Tcom’s usability testing was collected via a

survey. The survey was complemented with the current

testing process through four interviews and two

observations of usability tests, one internal and one

external. Parameters between the two tests were compared

and weighted against each other. The main findings reveal

that the internal process shows promising characteristics. It

is, up to, 1/10 as expensive per participant, has a quicker

call-to-test-time, facilitates transparency, and allows

refinement of the process. With the main findings in hand,

the conclusion points to the fact that usability testing should

be composed and conducted by Tcom itself. The greatest

challenge to such a change is how a usability perspective

can be integrated into the Scrum methodology.

Author Keywords

Product Development Process; User-Centered Design;

Usability Testing; Agile Framework.

ACM Classification Keywords

H.5.m. Information interfaces and presentation (Human-

Computer Interaction).

INTRODUCTION Product validation at any company is an important

component [20] in the Product Development Process

(PDP). Uncertainties and irregularities affecting this

process have a tendency of decreasing consistency and

usability throughout a product, most notably in cases where

a product is continuously iterated upon. There are various

types of methods to assess quality in products.

A shift towards an agile working process is under

implementation at the company. Although many aspects of

an agile environment are in place, there are areas that need

further attention. Such as the lack of usability attention

when developing products. Where an insufficient product

validation, in other words testing, is what this report aims to

investigate. Testing with users, which according to User-

Centered Design (UCD) methodology [26] concludes an

iteration as well as providing insights for the following

iteration. This perspective puts the users in center by

involving them in the process. UCD methodology with a

focus on usability is applied in this report since the author

perceives it can enhance product quality and due to a trend

of usability as sales arguments [4]. UCD is not yet fully

established in the PDP at Tcom and we asked ourselves

how difficult it would be to integrate usability testing with

their existing processes and whether the end result would be

better if the testing was done in-house as compared to being

done by others afterwards. In particular, we wanted to hear

the attitudes of the employees towards such a change. Thus,

part of the study presented here aims at investigating the

company culture and responses towards integrating such a

testing phase. Interviews were conducted with project

managers in order to understand the current testing process.

This method was complimented by a survey, where Tcom

employees’ perception of testing and the testing process

were in focus. The results indicated the prerequisites to

scale up the testing process, which is the next step for Tcom

in their pursuit of improving their PDP.

Two usability tests were studied. The former being a first-

ever in-house usability test executed at Tcom. The purpose

of this test was to investigate both a newly developed

purchase flow and the effectiveness of having this form of

testing in-house. Both tests will be described in greater

detail later in the report. The second usability test was

carried out by an external agency called Usbil, which is a

market research company that operates business-to-

business. They conducted a usability test ordered by an

employer organization called Mploy, where the purpose

was to test a set of scenarios in their new website. Usbil is a

Tcom partner, where they perform various types of testing

for the company. I took part in both respective testing

processes, acting as an active observer. As an active

observer I focused mainly on the process itself, while also

contributing with comments and insights to the test results.

Paste the appropriate copyright/license statement here. ACM now supports

three different publication options:

• ACM copyright: ACM holds the copyright on the work. This is the

historical approach.

• License: The author(s) retain copyright, but ACM receives an

exclusive publication license.

• Open Access: The author(s) wish to pay for the work to be open

access. The additional fee must be paid to ACM.

This text field is large enough to hold the appropriate release statement

assuming it is single-spaced in Times New Roman 8-point font. Please do

not change or modify the size of this text box.

Each submission will be assigned a DOI string to be included here.

Qualitative and quantitative data [13] was obtained through

the three earlier mentioned methods: online survey,

interview, and active observation. The questionnaire was

distributed in the company through two channels, for the

sole reason to increase the number of responses. The

interviews were held with four employees which resulted in

qualitative data regarding the current PDP. The two active

observations were included in order to procure hands-on

insight into the execution and post-process of usability

testing. In combination, these provided adequate data to

answer the research question:

- Can the PDP be improved by moving usability

testing in-house?

BACKGROUND

The background section presents the basics of UCD as well

as how UCD methods are applied in industry. Benefits and

different usability testing methods are furthermore

introduced. The section ends with further attention to the

two agile frameworks Scrum and U-Scrum.

UCD & frameworks The UCD process has a strong focus on involving the

human perspective in all its stages during its iterative

development cycle [26]. Which is essential for a computer-

based system to be used sufficiently by its users [10]. It puts

a central focus on insights guiding iterations until a desired

solution has been established. Insights are most often

generated from the likes of testing. UCD principles revolve

around: early focus on users and their tasks, evaluation and

measurement of product usage, and iterated design [26].

They can be decomposed further, but for the sake of this

report the key takeaway is that UCD is an ongoing process

with respect to users. Different methodologies are applied

by companies working with software development.

Whether it is Scrum [17], Kanban [12], RABBIT [26], U-

Scrum [22], or any other agile PDP the iterative design

process is to a great extent similar. All of them advocates

testing with users. But the testing method and execution

time of which differs. A plethora of testing standards can be

applied to accomplish the last stage of each iteration.

However, not all are suitable for every occasion and some

of the agile frameworks have not successfully integrated

HCD into the process [25]. Various parameters must be

considered in order to execute the most appropriate form of

testing.

Usability testing methods

When collecting input from (representative) users there is a

great number of methodologies to be used, to name a few:

A/B-testing [21], guerilla testing [16], usability testing [14],

and various statistical tools to map user behavior. Each of

these methods is acknowledged for its strengths and how

they suit different occasions, all will not be covered since

this work revolves around the process of usability testing.

Usability assessment, as defined by Murillo and colleagues

[14], is another way to distinguish usability testing from

other forms of testing. The assessment consists of two types

of methods: inspection methods and test methods. The

executor of the assessment decides which of the methods is

used. Inspection methods are performed by inspectors, most

often experts in the subject. Test methods, on the other

hand, engage representative users of the targeted customer

segment in a controlled environment, where usability

testing is a subcategory to these test methods. Depending on

which form of usability testing the observation differs, both

in terms of technology as well as the applied strategy of the

observers. Online or remote, usability testing utilizes the

opportunity of recording screen, video, and/or audio while

the user performs a test [1]. Face-to-face usability testing

has the same possibilities as well as the advantage of

interviewing the participant. A think-aloud method can

complement both types of usability testing [19].

Benefits of usability testing What makes the testing phase so important is the

opportunity to discover and map out errors and problems

perceived by the test participants [3]. Their input results in

insights which can pivot the following iteration, advocating

improvement. All errors cannot be acted upon, rather it

mainly depends on the magnitude and worth of fixing the

error, as Cockton argues in his research on Worth Centered

Design [2]. Nonetheless, usability testing provides

procedures to unveil problems that could harm products.

Usability testing is advised to execute in different stages

with different focus during the PDP [27] to ensure a high-

quality product. Most of the studied companies in [5]

perform usability testing in the stage before launch, to make

sure there are no vital problems. This approach may seem

as cost-efficient since the company only carries out one test.

However, since changes late in the PDP are costly, the

strategy is flawed. Last-minute fixes are also not a

guarantee that the product did in fact improve. What has

been suggested by the Nielsen Norman Group in their paper

[16], is how light-versions of usability testing can be

applied and generate important feedback with time- and

cost-efficient orientation. The main argument is that

usability testing can provide valuable insights without it

being conducted as a full-scale procedure.

Scrum The agile working method, mentioned earlier, is

furthermore implemented at Tcom utilizing the Scrum

methodology [6]. This framework is currently popular in

the software development world, where it has its

advantages of establishing a flexible and time-efficient

development process. The latter aspects of Scrum have

however not been fulfilled by a set of companies studied in

[5]. The problematics occur when the testing phase is

executed in the next coming sprint, and not in the current

one. This can result in insights from the test becoming

expired, and thus, are not relevant. The little attention to

usability in Scrum methodology is not surprising due to its

software development foci. The organization of Scrum

defines three roles: Product Owner (PO), Scrum team, and

Scrum master [4]. The PO is responsible for the product

backlog and deliverables. The development team can

include workers from many disciplines, they produce

deliverables through working in sprints. The Scrum master

makes sure that the development team can complete its

deliverables. This is a brief general view on Scrum

organization.

U-Scrum

Traditional Scrum fails to adequately address the usability

needs of the user [23]. Mona Singh applies a usability

perspective on Scrum, where she argues how Scrum lacks

the focus and resources to develop usable software

products. U-Scrum is suggested to enhance usability as it

applies such expertise on a high level in the Scrum team

organization through having two POs. The traditional,

single PO is complemented with another PO focused on

usability and User Experience (UX). This extension of

Scrum has not been adequately studied in current research,

but its takeaways can yield value for future agile

development teams.

METHOD Mixed Methods Research (MMR) approach has evolved

among researchers in recent years [9]. This way of

combining multiple forms of data in one research project

provides a deep understanding of context, experience, and

complexity [7]. A qualitatively driven approach was applied

for this report in order to cope with the complex nature of

usability testing at Tcom. Where qualitative data helped

shape the online survey as well as complementing the

quantitative data for a deeper understanding. This research

method enables comprehensive data collection of both

qualitative and quantitative characteristics [8]. Whereas

using only one method would have either supported a

detailed understanding of a rather niched aspect, or a wide

perception on a general level [8]. Therefore, this method

illustrates how accountability, budget, and opinions

influence the company culture in terms of in-house usability

testing. The empirical data was furthermore used to detect

any deviation of an objective reality. Each method has its

strengths and weaknesses, which is why results of the

survey and the interviews were put in relation to one

another, i.e. two methods validated each other’s

truthfulness. To deal with systematic errors during face-to-

face interviews [22] for example, the result of the

interviews is compared with results from other methods to

ensure its veracity. The three following methods were

utilized: online survey, observation, and interview. The

headquarter was the appointed office for this study since it

runs Tcom’s digital products.

Survey An online survey allows for fast distribution to a wide

audience, which made it the best possible tool for the

quantitative data collection since the headquarters of Tcom

holds 2200 employees and consultants ranging across

multiple departments distributed throughout an 8-story

building. The survey had a two-sided purpose in this data

collection: provide insight into employees’ attitudes

towards usability testing and ensuring interviews are in line

with the objective reality. Attitudes serve as a fundamental

argument for changing the current usability testing process

since employees are the ones utilizing processes and tools

in the work their opinion matter, which is similar to the

fundamentals of UCD.

However, the questionnaire designed for this thesis had to

take a wide set of respondents into account which affected

the technical terms and structure used. Survey participants’

answers will constantly be influenced by the phrasing of

questions and associated answer alternatives [22]. This

influence is possible to avoid up to a certain extent by

carefully structuring the order of questions. Where the most

important questions are advised to ask in the middle of the

survey, since respondents need priming before encountering

more cognitive demanding questions. While having

important questions late in the structure are likely to be

influenced by respondent mood, such as tiredness. Unlike

an interview, which can be slightly modified toward the

respondent in question [18], a survey must be the same for

every response.

Abstract concepts and detailed niche questions have a

tendency of confusing the respondent making her unable to

comprehend what is asked for [18]. This, in turn, results in

answers that do not reflect the question and increasing the

measurement error. Non-response is also a consequence due

to complex questionnaires. That is why an easily grasped

design and user-friendly approach was applied. Complex

subjects should be addressed by decomposing questions

into multiple elements, this has been explored in the

MultiTrait-MultiMethod (MTMM) [22]. This methodology

was appointed in the survey used for this report. Concept-

by-postulation is defined, by Willem E. Saris and Irmtraud

N. Gallhoferas, as the complex subject being investigated

[22]. This is achieved through exploring concept-by-

intuition, which is the subject in question decomposed to a

number of sub-subjects. These sub-subjects are additionally

converted into multiple measurable questions, defined as

request-for-answer. Request-for-answer will furthermore be

the terminology used throughout this report.

The survey generated both qualitative and quantitative data,

with more focus on the quantitative part. It included 31

requests-for-an-answer with the following structure: 3

straight open-ended requests, 4 multiple choice closed-

ended requests with an open-ended ‘other’ option, 12 single

choice closed-ended requests with an open-ended ‘other’

option, 2 multiple choice closed-ended requests, 10

multiple choice closed-ended requests. Both nominal and

ordinal [22] response scales were utilized. Where 4- and 6-

point symmetrical ordinal scales were applied when

requesting ratings. Saris et. al. suggests odd-number scales

to provide a neutral option for the respondents. However,

this was not implemented because of a will to encourage

respondents to take a stand and thus activating the

participants’ system 2 [11]. The nominal scales allowed

respondents to provide neutral answers, which were used in

the rest of the closed-ended requests.

The survey consisted of two separate flows: usability

testing and user testing. The requests were distributed over

6 sections, in the following order: a) respondent

background, b) experience with user/usability testing, c)

benefits of user/usability testing, d) user/usability testing

and Tcom, e) user/usability testing statements, f) Tcom in-

house user/usability testing. Section b) separated

respondents by knowledge to assign them into their

appropriate flow. Respondents familiar with usability

testing faced the term usability and were assigned to group

1, whereas the rest met the term user and were assigned to

group 2 or group 3. Group 2 had knowledge of user testing

but not of usability testing, and group 3 had knowledge of

neither. The data from the respective groups will be

analyzed and interpreted with respect to their size and

knowledge, i.e. group 1 is perceived as more significant

than group 3.

Together the 6 concepts-by-intuition generated company

attitudes towards testing, i.e. concept-by-postulation. The

online survey was distributed in two channels: Tcom

intranet and 90 posters put around the headquarter. The

posters were integrated with a link and a QR-code. The

intranet article generated 242 views whereas the Bitly-link

via the posters had 17 clicks. The posters around the

headquarters were not used for the response rate since one

could not identify how many of the roughly 2200 people in

the building noticed it. The QR-code had no tracker and

thus could not be used in calculations. Together the intranet

article and the Bitly-link served as the denominator for the

response rate of 29,3%. Although this response rate is

viewed as representative for the audience that accessed it.

An alternative response rate with responses / total number

of employees yields a less representative rate of 3,1%. Since

the survey targets opinions, the results can serve as a

guideline for Tcom employee’s attitudes, but not as

decisive.

Active observation of usability tests The two observations enabled an understanding of how

usability tests ordered by Tcom functions and how Tcom

could execute tests themselves. The two usability tests were

observed with a focus on the process. The following aspects

were studied: agenda, test environment, staff behavior,

participants, post-test effort, time, and expenses. Agenda

included features such as welcoming participants,

transcript, and test scenarios. Technology, furnishing, and

room layout were studied to understand which facilities

were available. The staff’s behavior was studied to

understand how the personnel acted throughout each test.

Whereas the participant gathering process revolved around

how attendants were chosen. The post-test effort category

focused on the efforts put in after the usability tests had

been conducted, where report compilation was of major

interest. Expenses for parts of the process and the whole

process formed the economical perspective. A time

perspective was used in three ways: time per test, time to

execute a usability test, and time until the report is

finalized.

Unstructured interviews Four unstructured interviews were conducted at the

headquarters with managers from different departments to

map the organization and the Tcom PDP. The first of which

was held with Product in order to understand how the PDP

works in practice, from idea to launch. Secondly, an

interview was conducted with Human Resources with the

aim of comprehending how the departments are distributed

and function alongside one another. The two described

interviews were conducted before the survey since that

qualitative data was a prerequisite for section a) and d) in

the survey. The third interview was held with an employee

from a department responsible for ordering usability tests

among other responsibilities. This interview provided an

understanding of how the testing process functions. Lastly,

the head of UX was interviewed to interpret how the testing

process functions. Additional interviews were not

conducted due to adequate data from the four interviews as

well as the restricted timeframe.

Except for having a main topic, unstructured interviews do

not follow a certain pattern [28]. Where the purpose is to

get as much qualitative data as possible through a

conversation with a user or stakeholder. Despite the number

of flaws with an unstructured interview [28] they serve as

an honest and investigative tool that results in an

understanding of opinions and experiences. A semi-

structured interview format would have resulted in more

structured in-depth data [15]. However, since it requires a

lot more effort to analyze the long transcripts it was not

assessed to fit the timeframe of the project.

RESULTS & ANALYSIS Data gathered with the three methods are organized

throughout the following section. Where it initiates with

presenting the survey respondent distribution followed by

the current usability testing process at Tcom. The latter is

complemented with relevant findings from the Usbil

observation. Whereas the Tcom in-house test presents how

a usability test could be executed. The section ends with

opinions towards usability testing and preferences regarding

a desired testing process.

Respondents 76 Tcom employees responded to the survey out of which

69 were located at the office of interest. The product

development is performed at this office and that is why the

7 employees from other offices were excluded. 47, or

68,1% of the respondents had previous experience from

usability testing and were assigned to group 1. Group 2

made up 13,0% of the respondents who had experience of

user testing but not of usability testing. Group 3 (18,8%)

had no experience with any testing methodology. The most

represented departments: IT Development (14%), IT User

Experience (14%), Network (13%), Product (12%), and IT

System Management (9%). Group and department

distribution are visualized in figure 1. Abbreviations used

for the departments are listed in table 1.

Figure 1. Department distribution of respondents.

Department Abbreviation Information Technology IT IT - Development IT D IT - QA IT QA IT - System Management IT SM IT - Architecture IT A IT - UX IT UX Customer Service CS Quality and Processes for CS CS Q Customer Relationship Management CRM Sales Sa Product Pr Communication Comm Business Intelligence BI Insight In Commercial Cml Marketing Ma Finance Fi Legal/Security L/S Business Transformation and Integration BT&I Human Resources HR Internet of Things IoT Technology Te Network Ne Customer Implementation CI Project Management PM

Table 1. Abbreviations of Tcom departments.

Current process

Interview The Tcom department structure is complex since it utilizes

the agile framework Scrum in parallel with a traditional

structure. The UX department is a good example of a

‘floating’ department since it functions dynamically in the

sense that its resources are applied in teams when

necessary. It is difficult to categorize all 2200 employee’s

positions in the building because not all departments work

agile. It also depends on how one defines the departments.

Which is further complicated by a recent company

reorganization. Scrum explains how the development teams

are organized but not the more traditional departments.

Although there is not one coherent way to successfully

divide all departments, the attempt used showed positive

results in the analysis of the quantitative data.

Currently, Tcom orders between two and three usability

tests each year from Usbil. The order-procedure is hugely

dependent on which employee(s) are involved in the

project. Hence, there is no established process for how to

request a usability test. The agile transformation which

Tcom initiated in early 2018 has been carried out in less

than one and a half years. The transformation has been

distressed by a reorganization in the company. The agile

environment is not completely fulfilled, which is

highlighted by gaps in the PDP. How to order a usability

test is one process that has been affected by this. The

process is described as ‘unclear’ and serves a lot of

competition from other processes in terms of budget and

attention. The result of this flaw is that usability testing

occurs rarely and late in the PDP.

When Tcom orders a usability test from Usbil, which is

their only partner in terms of testing, they regularly need

two weeks of preparation until the test can be conducted.

However, their fastest order-to-test saw a usability test

executed within a week from the date of the request. The

post-test effort from Usbil consists of a report that usually is

completed a week after the test. The total cost for these tests

is between 60-80 KSEK, most commonly 80 KSEK. The

expenses are bound to number of participants, participant

characteristics, post-test effort inter alia.

The absence of in-house usability testing at Tcom is mainly

due to budget restrictions according to the head of UX. The

resources are not enough at the moment to conduct more

usability testing. Tcom utilizes the following procedures of

UCD: personas (epics in Scrum), user journeys, expert

review, sporadic guerilla testing, and rare usability testing.

Guerilla testing and expert review are perceived as more

time- and cost-efficient than of usability testing.

Survey An average of 3,07 (out of 6) reveal that employees partly

disagrees with Tcom performing testing properly (Fig. 2).

26% of group 1 and 22% of the total respondents disagrees

(2) with the testing process at Tcom being performed

properly. Only 26% agrees (4 to 6) to some extent the

testing process is properly executed, whereas 74% rates the

process on the disagreeing part of the scale (1 to 3).

Figure 2. Tcom performs testing properly.

25% of the total respondents believe Tcom performs their

own testing. As mentioned above, this is not the case as

current external partner Usbil conducts testing for Tcom.

16% got the correct answer “no”, whereas 43% had no

opinion and 16% answered “I don't know”.

54% assess prototyping as the most important step in the

PDP (Fig. 3). 10 respondents had the opinion that testing

should be performed during each step in the PDP.

Figure 3. Which stage Tcom should test during the PDP.

Usbil observation The usability test, ordered by Mploy and held by Usbil, was

split into two days. Day one focused on the testing itself

whereas the second day worked through the notes during a

workshop. Usbil aimed for 5 tests having 7 participants

scheduled. One participant per hour was the predefined

testing rate, where each test was scheduled for the duration

of 45 minutes. 15 minutes in between the tests was reserved

for discussion within the crew regarding the participant, the

results, and any other comments.

The test room in which the moderator and test subject were

located in were equipped with microphones, video cameras,

a 4*2-meter glass-mirror, and an iPhone 6 used to run the

prototype. The smartphone-screen was live-streamed to a

monitor in the room behind the mirror where the rest of the

crew were located. The crew consisted of 4 Mploy

employees, me, and an Usbil employee. The moderator was

furthermore also employed by Usbil.

Each participant was welcomed, and a small chit-chat took

place. The participants were encouraged to present

themselves with name, age, work, and work role. Their

general knowledge of the topic in question was discussed

followed by specific Mploy questions. Each participant was

instructed to perform 4 scenarios. 5 minutes were reserved

at the end of the test for any further questions from the

crew. The test ended with the participants receiving their

compensation.

The crew had responsibility to catch everything of interest

on post-it-notes. They were furthermore put under either of

the four following categories: pain points, gain points,

about the participant, and other. The post-it notes were re-

categorized the next morning by the Usbil crew before the

workshop officially started. Sub-categories filled with

unique post-it notes were the major changes made.

The 3-hour workshop was structured as follows: refine

problem definition, construct a parking lot (thoughts that

might be paid attention), list quick-fixes, discuss actions

(main part of the workshop), and prioritize the actions. The

remaining effort of assembling a report is made by Usbil.

The average time for the report to arrive is 5 business days.

Mploy had stated certain requirements for the participants

that were recruited by Usbil. The more preferences, the

longer time Usbil need to acquire a minimum number of

participants.

Total cost for the 2-day usability testing session was 80

KSEK. Divided on 5 participants, the cost ended up being

16 KSEK per participant.

Tcom in-house usability test observation The first ever in-house usability test at Tcom was scheduled

during a full day, where a maximum of 10 participants

would attend, 9 showed up. Each test had a duration of 30

minutes with an additional 15 minutes reserved in between

tests. When recruiting participants, Tcom reasoned that they

aimed for at least 6 tests in total and thus booking more

than needed.

The testing took place at the company headquarters. A

laptop with the desktop adapted prototype was located on a

table, with a Tobii eye tracker mounted below the screen.

Video, audio, and the eye tracking movement were all live

streamed to another room, where the crew was located to

observe and analyze.

The moderator initiated the interview with a quick session

of friendly conversation before presenting the agenda of

their upcoming 30 minutes. Once the participant had agreed

to be recorded, the moderator lead the participant into the

subject by asking a few test-oriented questions about the

participants’ situation. The moderator put emphasis on the

fact that the participant is not being tested, it is the

prototype. The test consisted of three scenarios.

The crew in the observer-room thoroughly studied the eye-

movements, comments, and body language of the

participants on a 42-inch screen. The crew consisted of a

floating number of employees, at least three employees

were present for each of the tests. Notes were generated

based on anything which the crew felt appropriate. Which

was shared and discussed in the room after each test, with

the moderator present. Thoughts, tips, and manuscript

comments were continuously shared to iterate and improve

the test.

Once the 9 tests were completed, the following procedure

took place: combine and summarize notes, draw

conclusions, create a report. Dropbox Paper was utilized

throughout this process, where we were 4 persons in total

that contributed to the post-test effort. The report was

compiled after three business days.

Tcom appointed Rcru to recruit participants for the usability

test. Specific requirements can be accomplished by the

recruiting company. In this case, Tcom desired participants

based on two requirements. The more requirements, the

longer time Rcru need.

Total expenses for the Tcom usability test was 15,5 KSEK.

The expenses include recruitment costs and compensation.

Cost per participant ends in 1’722 SEK.

Tcom employees’ opinions towards testing 98,6% of the respondents believe testing is beneficial/partly

beneficial on a scale from 1 (not beneficial) to 4

(beneficial). Where 65,2% rated it as beneficial. The three

groups agree to a large extent about testing being beneficial,

where the exception is group 3 in which 46% consider

testing to be partly beneficial and 46% beneficial.

“Doing the right things” was a common response in the

open-ended feedback. Additional comments: “finding

errors”, “mitigate risk”, “to see if you are on track”,

“understand how the product is used”, “gives insights that

are impossible to predict”.

The primary benefit of testing stated by group 1 and group

2 is to study customer behavior, whereas group 3 ranks

error discovery as the primary asset of testing. Validating

assumptions is considered as the second most beneficial

attribute from the respondents familiar with usability testing

(group 1). This is however not shared by the other two

groups, where study customer behavior and error discovery

are those group’s top choices.

88,4% believes testing contributes to the quality of products

when ranked from 1 (nothing) to 6 (essential contribution),

shown in figure 4. Group 3 is slightly skeptical towards

rating the contribution higher than 4 (moderate

contribution). In average, group 2 were most optimistic

about testing contributing to product quality with an

average score of 5,1. Followed by group 1 with an average

of 4,9, whereas group 3 indicated more skepticism in

relation to the other groups with an average of 4,5.

Figure 4. Testing contributes to product quality.

Request-for-answers about continuous testing during PDP

resolved in an average score of 5,39. The scale in figure 5

was as follows: 1 (strongly disagree) to 6 (strongly agree).

Where 59% of the respondents strongly agreed continuous

testing should be part of the PDP. 5 (agree) and 4 (partly

agree) were the second and third most common answers

with 25% and 12% respectively. Group 3 returned the most

pessimistic result with an average score of 4,69.

Figure 5. Testing should be continuous during PDP.

Two-thirds of the respondents strongly agreed with testing

helps increasing product quality. An average of 5,51

revealed almost no negative opinions towards the relation

of testing and product quality. Group 1 indicated the highest

average of 5,62, whereas group 3 showed a more

discouraging relation with an average of 5,08.

The importance of testing saw respondents across the three

groups agree, indicated by an average of 5,41. 62% strongly

agreed (6), 20% agreed (5), and 13% partly agreed (4) with

testing being important.

Tcom employees testing process preferences There is nothing stopping Tcom from introducing a test-day

every other week. Where a strong will amongst the

employees is to move usability testing in-house. There are

however a few unresolved aspects that obstruct in-house

testing: ownership, budget, time, participant gathering,

compensation, and administration. Many of which are

admittedly plausible to resolve, where the most significant

facet, ownership, is where the many discussions have come

to a halt. Expertise is not considered as an issue since UX

and Insight both obtain the skills needed.

There are aspirations of having an in-house testing process

at Tcom where an employee responsible for the whole

testing process is suggested. Consistency is a concern if an

in-house process would be realized.

Tcom was rated as the number one company to perform

testing for its products followed by Tcom’s current external

partner Usbil. Which received nearly half the votes in

relation to Tcom. Other external partner showed a similar

result as Usbil. 62% of the respondents belonging to group

3 opted no opinion, which is higher than for the other two

groups.

One day is considered as 8 hours. The average testing time

per week lies somewhere between ½ day and 1 day (Fig. 6).

4 respondents stated that the testing time per week

“depends on the product” and 46% had no opinion in the

subject. Where no opinion distribution was as follows: 36%

of group 1, 78% of group 2, and 62% of group 3. A

majority stated that the time needed depends on the

complexity and the magnitude of the product in question.

Some of the respondents indicated that the time needed

should make up 20-25% in relation to the total time spent

on the product. “More tests - better products” was also

mentioned.

Figure 6. Average testing time per week.

61% of the respondents believe Tcom should perform their

own testing. Whereas 16% are against that idea and 23%

had no opinion. 5 answers were excluded due to

inconclusive format “Both”. 70% of the most

knowledgeable respondents (group 1) are of the opinion

that Tcom should perform testing themselves. Respondents

who answered yes (39) could answer the two follow-up

questions regarding who should perform and who should be

responsible for in-house testing at Tcom.

UX (31%) and Product (13%) are thought to be the two

departments that should be conducting in-house testing.

10% of the respondents had no opinion in the matter.

Figure 7. Which Tcom department to conduct in-house testing.

36% of Tcom employees believe Product should have the

responsibility of in-house testing. Insight (15%) and UX

(13%) are the two following options stated by the

employees.

Figure 8. Which Tcom department to be responsible for in-

house testing.

The primary benefit of in-house testing is not mutual

between the three groups. Opportunity to refine the process

was ranked as group 1’s first choice of benefit with 30%.

Whereas group 2 assessed more frequent testing and control

in the process as the two most important benefits with 33%

of the votes respectively. Group 3 considered cost-

efficiency as the greatest impact of in-house testing. 38% of

group 3 had no opinion in the subject. Three benefits stand

out when the primary benefit was requested: opportunity to

refine the process (1), control in the process (2), and cost-

efficiency (3) (Fig. 9).

Figure 9. Top 5 benefits with in-house testing at Tcom.

All groups agreed bias may influence the results is the main

drawback of in-house testing (42%) if we ignore the 46% of

group 3 which had no opinion. “Demands too much time”

is the second rated primary drawback with a total of 28%.

Only a few (6%) assessed no drawbacks of in-house testing.

Figure 10. Top 5 drawbacks with in-house testing at Tcom.

DISCUSSION

The discussion revolves around the findings in result &

analysis and is structured in a similar way. It initiates with

the current testing process, followed by Tcom employees’

opinions towards usability testing. Then follows a

comparison between the two observed usability tests. The

section ends with employees’ preferences on the process of

usability testing.

Current testing process Ownership, budget, time, and administration are the main

factors restricting an in-house usability testing process.

Although most of the employees believe an in-house

process is a more efficient solution, the process falls in

between departments as it is structured now. Insight lacks

the resources to be responsible and to carry out the testing.

Although UX employees have the mandate to request

usability tests, the current go-to for executing a usability

test demands a lot from the requester as they are required to

lead the whole process which results in few tests.

A clear majority of the respondents (74%) believes Tcom’s

testing is inadequately performed. The few numbers of

annual usability tests, as well as a vague order-process,

have raised these opinions around the agile departments.

The prototyping stage is considered as the most vital stage

to test. When testing occurs at Tcom, it is during this stage.

It can be dangerous to stall tests until the prototyping stage

since decisions on the fundamental aspects of the project

have been made without solid validation. Hence, expensive

adjustments may have to be carried out in order to make

sure products are of the desired quality. Scrum

methodology is implemented to avoid these incidents, but

since Scrum mostly concerns the software development

rather than usability these incidents can still occur.

Tcom employees’ opinions towards usability testing Respondents in all three groups consider testing as

important, beneficial, and that it increases product quality.

Testing should be conducted between four and eight hours

per week according to employees. The answers indicate

differences in their perception of testing showing a pattern

descending with experience of user- and usability testing,

i.e. employees with much knowledge rates testing as more

beneficial and vice versa. Respondents in group 3 consider

a transparent process as the most beneficial aspect with in-

house testing, which could originate from a lack of insight

into Scrum teams progress.

Bias and expertise in an in-house process is a concern

among respondents. The expertise exists as observed in the

active observation of Tcom’s in-house usability test and

stated during two of the interviews. Skewed results from in-

house testing are however not believed to be a concern

according to an Insight manager. Some of the respondents

furthermore considered “the ones closest to the project” as

best suited to conduct testing. Insight can assist with

moderators for tests with no insight into the PDP and thus

providing an objective approach.

Comparison between in-house and Usbil test Tcom in-house tests show promising costs of 1/10 per

participant in relation to Usbil. Although this ratio does not

include expenses for Tcom employees time and efforts, the

price tag is still favorable. The execution time is not

advantageous compared to Usbil unless the process is

changed. Where a standing test day every other week could

be the solution if time-efficiency is to be enhanced in terms

of call-to-test-time. There is another aspect of time-

efficiency which revolves around the post-test effort and

go-to steps based on insights during the tests. Usbil requires

one week to compile a report, which is inefficient compared

to Tcom’s three business days. The report is furthermore

not as important as one might expect, the go-to steps are the

reason for usability tests to be conducted in the first place.

Which can be extracted directly from the tests and avoiding

a report, and thus reducing the time until changes in the

product are applied. Usbil’s usability test procedure is

however more organized than Tcom’s, as it follows a

certain framework, which is likely to generate more on-

point insights.

Tcom employees testing preferences Employees have expressed their willingness to have a

greater insight into the advancement of projects. Which can

be realized via inviting squad members to observe tests as

well as through the possibility to stream in-house tests.

The benefits and gains that can be achieved when testing

early in the process cannot be pursued with the current

testing process due to scarce recognition of usability. An in-

house testing process is believed to proclaim more testing

during different stages in the PDP. Expenses can

furthermore be reduced if the project validates its

advancement from the get-go. Although a range of

limitations are holding in-house usability testing back,

employees have expressed its potential of improving the

PDP. Continuous usability testing has been requested for a

few years but has served resistance from the current

framework Scrum.

Nature of usability testing

The usability testing methodology studied in this work is

one of the many formats’ customer input can be obtained.

Various methodologies and technologies can be applied for

the sake of validating and studying prototypes and products.

The in-house testing process shows the most prominent

characteristics of the two, but there are further alternatives

who can bear an equal or perhaps better process. A

compromise between an in-house and external process is

one example, where the test is conducted through a near

collaboration with external and internal experts at an off-

site lab. Another procedure could be how a company

utilizes external experts in an in-house process. The main

argument here is that there are several ways in which

usability testing can be organized and that this work is just

one of the many ways’ usability testing can be conducted.

Like the data collection method used in this work (MMR),

companies do not solely rely on one product validation tool.

Usability tests are usually complemented with quantitative

statistics such as heat maps or drop-out rates. They can be

further complemented with A/B-testing, where two versions

of a product/prototype are released and compared against

each other. Customer feedback during the customer journey

is a voluntary instrument which can help companies in

understanding their products. Multiple tools will strengthen

the empirical data and thus increasing the accuracy of the

analysis.

U-Scrum

The usability focus of U-Scrum is thought to provide

fundamental aspects in the product lifecycle, foremost

during the early stages in order to understand the users and

their needs. Once the business case is mapped the need for

two POs could thus decrease and the organization is back to

having solely one PO. One PO would resolve in distinct

leadership whereas two of them could perhaps trigger a

stalemate situation where focus would turn to who is the

“true” leader instead of the methodology’s main purpose.

Such a situation is likely to harm the PDP. In contrast, two

POs have the opportunity to focus on different aspects

throughout the process resulting in efficient management.

The main concern against two POs is when they both are

involved in the same aspect with different opinions, who

will obey? This question is left unanswered in today's

research and will need further attention to be dissolved.

CONCLUSION Tcom initiated their implementation of Scrum in early

2018, where the agile working method is applied but has an

acknowledged flaw in terms of usability attention.

Employees perceive usability testing as scarcely performed.

Budget motives and ownership restrains testing due to other

processes being prioritized. As a result, usability testing

occurs twice a year on average. Tcom has little control in

the current testing process since Usbil takes care of the

process once the order is made, which is expressed as

opaque by some employees in the Scrum teams.

- Can the PDP be improved by moving usability

testing in-house?

In-house usability testing could conduce to more testing

since it is far cheaper. The expense per participant from the

in-house pilot test was roughly 1,7 KSEK, that is a close to

1/10 of Usbil’s cost per participant. Tcom will also have

total control in the process which enables iterative

improvement of the process itself. Where time-efficiency

shows great potential, both call-to-test-time and time spent

on post-test effort can serve enhancement. The ability to

provide a more transparent process is another advantage

with usability testing conducted in-house. Although bias

and expertise were stated as concerns of in-house testing,

Tcom currently holds the expertise to manage those

concerns. As a result, improving the usability testing

process will give greater insight into products and thus

advocating improvement of the PDP.

Another finding during this study was how the agile

framework Scrum deals with the concept of UCD and

usability. Personas, user journeys, and expert reviews are

utilized in Tcom’s PDP. Conclusions from those methods

are not validated through testing with users, where one must

question if Scrum is the most appropriate framework from a

usability perspective.

FUTURE WORK

In order to introduce an in-house usability testing at Tcom

there must be an executive decision made regarding which

department(s) should be involved in the process as well as

setting aside a budget. Exploring U-Scrum methodology is

possibly one solution for this issue. Once the responsibility

is resolved Tcom can initiate structuring the process and

investigating which partners to work with. The integration

of a usability testing process will presumably see a trial

period before it is fully implemented.

ACKNOWLEDGMENTS

I would like to thank Tcom for taking me in and giving me

the opportunity to be a part of their team and allowing me

to explore their PDP. It has been a pleasure to have worked

side-by-side with such an enthusiastic and welcoming set of

individuals. A special thanks go to my supervisor and my

stand-in-supervisor at Tcom.

REFERENCES

1. Albert, B., Tullis, T., & Tedesco, D. 2010. Beyond the

usability lab : conducting large-scale user experience

studies. Beyond the Usability Lab.

2. Cockton, G. 2006. Designing worth is worth designing.

In Proceedings of the 4th Nordic conference on

Human-computer interaction: changing roles (pp. 165-

174). ACM.

3. Davis, K., & Horst, D. 2012. The note-taker’s

perspective during usability testing: A hands-on

approach to recognizing what’s important, what isn’t.

In Proceedings of the Human Factors and Ergonomics

Society. pp. 2412–2416.

4. Düchting M., Zimmermann D., Nebe K., 2007.

Incorporating User Centered Requirement Engineering

into Agile Software Development. In: Jacko J.A. (eds)

Human-Computer Interaction. Interaction Design and

Usability. HCI 2007. Lecture Notes in Computer

Science, vol 4550. Springer, Berlin, Heidelberg.

5. Eloranta, V.-P. et al. 2013. Scrum Anti-Patterns -- An

Empirical Study. 2013 20th Asia-Pacific Software

Engineering Conference (APSEC), 1, pp.503–510.

6. Gonçalves, L. 2018. Scrum. Controlling &

Management Review, 62(4), pp.40–42.

7. Halcomb, E. & Hickman, L. 2015. Mixed methods

research. Nursing standard (Royal College of Nursing

(Great Britain): 1987), 29(32), pp.41–47.

8. Hall, Jori N. & Ryan, Katherine E. 2011. Educational

Accountability: A Qualitatively Driven Mixed-

Methods Approach. Qualitative Inquiry, 17(1), pp.105–

115.

9. Hunter, B. 2009. Advances in Mixed Methods

Research: Theories and Applications. Drug and

Alcohol Review, 28(5), p.575.

10. Issa T., Isaias P. 2015. Usability and Human Computer

Interaction (HCI). In: Sustainable Design. Springer,

London.

11. Kahneman, D. 2011. Thinking, fast and slow 1. ed.,

New York: London: Farrar, Straus and Giroux; Allen

Lane.

12. Leopold, K., & Kaltenecker, S. 2015. Kanban Change

Leadership: Creating a Culture of Continuous

Improvement. Somerset: Wiley.

13. McLeod, S. A. Qualitative vs. quantitative research.

2017. Retrieved May 21, 2019 from

https://www.simplypsychology.org/qualitative-

quantitative.html

14. Murillo B., Vargas S., Moquillaza A., Fernández L.,

Paz F. 2017. Usability Testing as a Complement of

Heuristic Evaluation: A Case Study. In: Marcus A.,

Wang W. (eds) Design, User Experience, and

Usability: Theory, Methodology, and Management.

DUXU 2017. Lecture Notes in Computer Science, vol

10288. Springer, Cham.

15. Newcomer, K.E. 2015. Conducting Semi‐Structured

Interviews. In Handbook of Practical Program

Evaluation. Hoboken, NJ, USA: John Wiley & Sons,

Inc., pp. 492–505.

16. Nielsen, J. 1994. Guerrilla HCI: using discount

usability engineering to penetrate the intimidation

barrier. In Cost-justifying usability, Randolph G. Bias

and Deborah J. Mayhew (Eds.). Academic Press, Inc.,

Orlando, FL, USA 245-272.

17. Opelt, A. 2013. Agile contracts creating and managing

successful projects with Scrum, Hoboken, N.J.: Wiley.

18. Ornstein, M. 2013. A Companion to Survey Research,

London: SAGE Publications Ltd.

19. Paz, F. & Pow-Sang, J.A. 2015. Usability Evaluation

Methods for Software Development: A Systematic

Mapping Review. 2015 8th International Conference

on Advanced Software Engineering & Its Applications

(ASEA), pp.1–4.

20. Quiñonesa, D., Rusua, D. & Rusub, V. 2018. A

methodology to develop usability/user experience

heuristics.

21. Rubin, J., Chisnell, D., & Dawsonera. 2008. Handbook

of usability testing how to plan, design, and conduct

effective tests (2nd ed.). Indianapolis, IN: Wiley Pub.

22. Saris, W.E. & Gallhofer, I.N. 2014. Design,

Evaluation, and Analysis of Questionnaires for Survey

Research, 2nd Edition, John Wiley & Sons.

23. Singh, M. 2008. U-SCRUM: An agile methodology for

promoting usability. In Agile 2008 Conference (pp.

555-560). IEEE.

24. Siroker, D., Koomen, P., & Harshman, C. 2013. A/B

testing: The most powerful way to turn clicks into

customers.

25. Sohaib, O., & Khan, K. 2010. Integrating usability

engineering and agile software development: A

literature review. In 2010 international conference on

Computer design and applications (Vol. 2, pp. V2-32).

IEEE.

26. Still, B., & Crane, K. 2017. Fundamentals of User‐

Centered Design: A Practical Approach. Boca Raton,

FL: CRC Press.

27. Sy, D. 2007. Adapting usability investigations for agile

user-centered design. Journal of usability Studies, 2(3),

112-132.

28. Wilson, C. 2014. Interview techniques for UX

practitioners: A user-centered design method.

www.kth.se

TRITA -EECS-EX-2019:561