Comparing the Effectiveness of

Electronic Diary and UX Curve Methods

in Multi-Component Product Study

Farrukh Sahar

Jari Varsaluoma

Sari Kujala

Contents

• Introduction

• Motivation and problem statement

• Methods

• Results

• UX Evaluation Implications for UX

Practitioners

• Conclusion

4.11.2014 2

Introduction

• In this study, the goal is to measure the effectiveness of

an electronic diary and the UX Curve method as remote

research methods for collecting qualitative data.

• The cost-effectiveness of the methods was evaluated

using two factors: the amount of time required by 1) the

researchers in employing the methods and 2) the

participants to report their experiences with the method.

• Also, to determine the quality of the reported

experiences, their content was compared between the

methods.

4.11.2014 3

Motivation and problem

statements

• in the literature there is lack of research on measuring

and comparing the effectiveness of electronic diary and

the UX Curve method as remote research method in the

long-term UX evaluation.

• In addition, there is a need to measure and compare the

effectiveness of methods in order to guide UX

practitioners on selecting an appropriate and effective

research method that enables valid and reliable

evaluation of long-term UX.

4.11.2014 4

Methods and studied

product

• 25 participants

• we conducted an eight-week-long electronic diary

study and subsequently employed the UX Curve

method.

• The studied sports watch system had four components.

The sports watch was a main component with three

other components: 1) a speed sensor; 2) a heart rate

belt; and 3) a sports website. All the participants had

bought the products for themselves and had been using

the product at least five times per week.

4.11.2014 5

Sports watch system

4.11.2014 6

Electronic diary and the UX

Curve method

4.11.2014 7

Electronic diary and the UX

Curve method

4.11.2014 8

Results=>Positive and

negative experience items

• In the positive experience items’ utility subcategory, the

participants reported more utilities related to the sports

watch system during the electronic diary study than with

the UX Curve method.

• Interestingly, there were 11 stimulation category

experiences related to the sports watch that users

reported only with the UX Curve method. Furthermore,

there were 12 stimulation category experiences related

to the sports website that users reported only with the

UX Curve method.

4.11.2014 9

Results =>Positive and

negative experience items

• In addition, the findings suggest that the electronic diary

was more effective than UX Curve method in capturing

the missing functionalities and technical faults.

• The findings show that even though the structures of

both methods were different, both methods captured

more experiences related to sports watch and sports

website compared with speed sensor and heart rate belt.

4.11.2014 10

Results =>Cost-effectiveness

• Comparison of cost-effectiveness of the electronic diary

study and the UX Curve method

4.11.2014 11

Results =>Cost-effectiveness

• Comparison of cost-effectiveness of the electronic diary

study and the UX Curve method

4.11.2014 12

Results =>Cost-effectiveness

• Comparison of cost-effectiveness of the electronic diary

study and the UX Curve method

4.11.2014 13

Results =>Cost-effectiveness

• Comparison of cost-effectiveness of the electronic diary

study and the UX Curve method

4.11.2014 14

Results =>Cost-effectiveness

• Comparison of cost-effectiveness of the electronic diary

study and the UX Curve method

4.11.2014 15

• With the electronic diary study the participants reported

their experiences in more detail than with the UX Curve

method.

• In our electronic diary study motivating the

participants and getting a good user response rate was

a challenge for the researchers because participants

were required to fill the weekly diary for two months. On

the other side, the UX Curve method required less

efforts by the researchers because participants were

required to draw the UX curves only once at the end of

the study.

4.11.2014 16

Results =>Cost-effectiveness

Results =>Workload

• During the UX Curve method data analysis, the

researchers invested efforts in deciphering the

handwriting of participants and transferring the data into

the electronic format.

• We experienced that because of the huge number of

details in the 200 collected diaries the data analysis

process was extensively laborious in nature and

consumed more time than the UX Curve method.

4.11.2014 17

UX Evaluation Implications

for UX Practitioners

1) In order to effectively reveal utility related experiences,

the UX practitioners should use an electronic diary

study rather than the UX Curve method.

2) In order to capture the missing functionalities and

technical faults, the UX practitioners should use an

electronic diary study rather than the UX Curve method.

3) The UX practitioners could use an electronic diary study

instead of the UX Curve method to reveal experiences

related to dissatisfaction.

4.11.2014 18

UX Evaluation Implications

for UX Practitioners

4) In order to reveal experiences associated with

effectiveness, the UX practitioners could use an electronic

diary study instead of the UX Curve method.

5) It seems that the UX practitioners could use the UX

Curve method rather than an electronic diary study in order

to capture the positive experiences related to stimulation.

4.11.2014 19

Conclusion

• Our results suggest that both methods are effective in

revealing specific type of information and have their

strength and weaknesses.

• The electronic diary was more effective than the UX

Curve method in revealing utilities, missing

functionalities and technical faults related to sports

watch system.

• On the other hand, the UX Curve method was more

effective than electronic diary in capturing the positive

experience items related to stimulation category.

4.11.2014 20

Limitations and Future

Studies

• The reported experiences with the electronic

diary study could have affected on the

remembered experiences with the UX Curve

method.

• In the future, it is important to study the

effectiveness of other methods during the

remote UX evaluation of multi-component

products.

4.11.2014 21