usability test results xtext new project wizard

© itemis AG

Findings from Usability TestsTested application: Xtext New Project Wizard

Date of test: 04.12.2016, Version: Xtext since v2.9

© itemis AG

Summary

© itemis AG

• Reason for the usability tests was the discussion in GitHub: [wizard] Improve usability of Generic IDE Support option #119https://github.com/eclipse/xtext-eclipse/issues/119

• Goal of the test was to compare three different possible solutions of the advanced Xtextconfiguration wizard. The solutions were created as mockups.

• We conducted usability tests to get feedback regarding the three solutions:

• 5 testers performed different tasks with the configuration editor

• Duration of the tests: 15 minutes

Usability findings for Xtext New Project Wizard

Summary

3

https://github.com/eclipse/xtext-eclipse/issues/119

© itemis AG

Version 1 – Original wizard

The option „Generic IDE Support“ ismandatory for the options „EclipsePlugin“, „IntelliJ IDEA Plugin“ and „Web Integration“.

If one of the three options is selected, a hint is shown with a quick fix to enable„Generic IDE Support“. The user must either click on the quick fix link in the hintor select the checkbox.

4

© itemis AG

Version 2 – Automatic selection


If one of the three options is selected, „Generic IDE Support“ is directly selectedand the checkbox disabled. A hint isshown explaining why.

5

© itemis AG

Version 3 – Hierarchies


Thus, „Generic IDE Support“ it is at higherlevel. The three checkboxes below areonly enabled if „Generic IDE Support“ isselected.

6

© itemis AG

General:

Version 2 was rated best

Version 1 was rated worst

Three most important findings:

• With version 1 and 3 the most usability problems occurred

• The current interface structure is error-prone as the hint does not really support to see the dependency between “Generic IDE Support“ and the options „Eclipse Plugin“, „IntelliJ IDEA Plugin“ and „Web Support“.

• Meaning of terms „Generic IDE Support“ and „Source Layout“ was not clear

Overview on results

Summary

7

© itemis AG

Study design

© itemis AG

ID Experience with Xtext Experience withProject Wizard

P1 Low. Took part in trainings but has not worked on an ownproject. None.

P2 None. None.

P3 None. None.

P4 Medium. Used it some years ago. None.

P5 Medium. Used it some years ago. None.

Participants

All participants are developers

9

© itemis AG

First, the testers got the original version of the wizard and were asked what the different terms mean. Afterwards they had to rate how sure they are regarding their answer.

Then they performed a task with all versions (sequence of versions varied). We used the „Think Aloud“ method: all participants explained what they do and think while executing the task.

Tasks (with each version):

• „Please configure a new Xtext project. Please assume for the following task that you want tocreate a new Xtext project that includes support for an eclipse editor frontend.“

• Questions asked after performing the task with a version: - What was your impression regarding the dialogue?- Does the dialogue support you in choosing the right options?

• Post session questions (after all three versions were tested):

• Which dialogue supports you best in fulfilling the task? Why?Which dialogue was worst in supporting you? Why?Which suggestions for improvement do you have?

Use of New Project Wizard

Test scenario

10

© itemis AG

Results

© itemis AG

Severity

Cosmetic – inconsistencies, visual problems, problems with content

Positive – What was good, easy, intuitive

Minor – confused user, task cannot be solved fastly

Major – user has problems to fulfill the task, needs a lot of time and steps to reach goal

Blocker – user cannot solve task, cancels tasks

12

© itemis AG

1.: What is the meaning of…

© itemis AG

Version 1 – Meaning of „Eclipse Plugin“

Answer Correctness How sure are you? (1-6; 1 = very sure) ID

An eclipse version Wrong 4 P1

You get an eclipse plugin based on what you didwith Xtext. It is available in the runtime. Correct 1 P2

Means that I want to include my Xtext project in eclipse. Correct 2 P3

You can create an eclipse plugin that you can re-install in another eclipse IDE. Correct 2 P4

The eclipse IDE integrated Xtext plugins will begenerated. Correct 1 P5

Most users are very sure regarding their answer and answered correctly.

14

© itemis AG

Version 1 – Meaning of „IntelliJ IDEA Plugin“


Configuration and use of the project in IntelliJ (or access to the project) Partly 3 P1

You get an IntelliJ plugin based on whatyou did with Xtext. It is available in theruntime.

Correct 1 P2

Means that I I want to include my Xtextproject in IntelliJ. Correct 1 P3

Extention for IntelliJ that can beintegrated. Correct 2-3 P4

The necessary Glue code (probably in addition to core plugin) will be generatedto integrate it in IntelliJ.

Correct 1 P5

Most users are sure regarding their answer and answered correclty.

15

© itemis AG

Version 1 – Meaning of „Web Integration“


Integration of Xtext in a website. Partly 4 P1

You generate components (client andserver) that can be used in a web context.

Correct 1 P2

Integration of some kind of web access. Correct 2 P3

You can use the DSL that isimplemented also in web. Correct 2 P4

An orion based editor will be created. Partly 2 P5

Most users are sure regarding their answer. They answered correctly or partly correct.

16

© itemis AG

Version 1 – Meaning of „Generic IDE Support“


No idea -/- 1 P1

Not sure, maybe a third type of IDE or forruntime without IDE Wrong 5-6 P2

That it is supported from my environment. Don‘t know how to describe. Wrong 3-4 P3

No idea. -/- 1 P4

Connection to language server protocol ora generic editor. Correct 3 P5

Meaning of „Generic IDE Support“ is not clear for participants: either they had noidea or they were not sure regarding their answer.

Problematic as this is mandatory for some of the other possibilities

17

© itemis AG

Version 1 – Meaning of „Testing support“


Include a test runner or a test environment. Correct 3 P1

Test classes for developed DSLs. However, tests are always necessary so not sure whyto have it as separate aspect.

Correct 5 P2

To see how my project is tested and that I know who tested what when. Wrong 2 P3

Provides JUnit (etc.) with different testcases. Wrong 3 P4

Creation of test bundles in addition toproductive bundles. Correct 2 P5

Users are not really sure what this means and only some answered correctly.

18

© itemis AG

Version 1 – Meaning of „Preferred Build System“


Somethink such as Gradle, Maven, Ant. Correct 2 P1

Maven etc to generate the correspondingartifacts. Correct 1-2 P2

That I can choose between the different systems in the dropdown when I build myproject.

Wrong 4 P3

Maven, Ant, Gradle Correct 2 P4

Which build system I will use (e.g. onlybuild it in eclipse or with Gradle or Maven). Correct 1 P5

Most users are sure regarding their answer and answered correctly.

19

© itemis AG

Version 1 – Meaning of the terms: „Source Layout“


Components that should occur in a project(alternatively, you get an empty project). Source layout is the file in which you do theconfiguration.

Wrong 2 P1

Structure of the source code. Wrong 4 P2

If the interface of the IDE is multilayered orsingle-layered. Is necessary to configure theproject hierarchy.

Wrong 4 P3

Not sure if this affects generated code or thesource that describes the DSL. Wrong 4 P4

Don‘t know if this concerns the build systemor the eclipse project or some content. Wrong 2 P5

Most users are not really sure what this means. All answered wrong.

20

© itemis AG

2.: Which dependencies do you seebetween the different options?

© itemis AG

Version 1 – Dependencies between options

Answer How sure are you? (1-6; 1 = very sure) ID

Not really sure. I do not understand the dialog in a way that somefacets exlude others. I think I can use them idenpendently. 1 P1

It depends if the selected IDEs support a specific build system. The other options are independent. -/- P2

Create Feature and Create Update Site depend on EclipsePlugin. For the other options I cannot see that there aredependencies.

1 regarding Create…5 for rest P3

Create Feature and Create Update Site depend on EclipsePlugin. For the other options I cannot see that there aredependencies.

1 P4

IntelliJ and Web Integration are independent from eclisepossibilities. Not sure if core plugins are also generated asEclipse plugins if I will not choose Eclipse plugin.

3 P5

The current interface structure does not show that „Generic IDE Support“ is mandatory for the options „Eclipse Plugin“, „IntelliJ IDEA Plugin“ and „Web Support“

22

© itemis AG

Issues concerning version 1 - Original

Didn‘t expected dependencies between„Eclipse Plugin“ and „Generic IDE Support“ because of layout (P2, P5).

Does not like that he has to click on the link in the error message (P2).

Expected that „Generic IDE Support“ isselected automatically (P2).

Sees hint but was not sure if this means thathe has to check „Generic IDE Support“ (P3)

Liked that the checkmark was not setautomatically and that you have the choice(P4) -> however, didn‘t understood thatthis is necessary

24

© itemis AG

Issues concerning version 2 – Automatic selection

Liked that checkmark for „Generic IDE Support“ is created automatically(P1, P3, P4, P5).

Likes that there is no unnecessary hintsaying what he has to do (P3).

Good that mandatory option cannot bedeactivated (P5).

Structure does not show dependency (P5).

Hint should be placed directly at „GenericIDE support“ when it is disabled (P5).

25

© itemis AG

Issues concerning version 3

Wanted to select „Eclipse Plugin“ but was not possible as it was deactivated („Generic IDE Support“ was not selected). Was confusing(P1, P2)

The layout implied the tester that he must select one of the three sub-options, too andthat it is not possible to use „Generic IDE Support“ only (P2)

Layout shows dependency between„Generic IDE Support“ and „Eclipse Plugin“ (P3, P4, P5)

Layout shows that there is no dependencybetween the IDE issues and Testing Support (P5)

Tester didn‘t see „Eclipse plugin“ because itwas disabled first (P3)

Unclear structure compared to otherdialogues (P4).

Two clicks are necessary to activate „EclipsePlugin“ (P4)

26

© itemis AG

Which dialogue is the best?

Answer ID

Version 3. However, I also liked version 2 as necessary options are automatically selected. P1

Version 2 P2

Version 2 but version 3 was also good. P3

Version 2. Checkmark was done automatically and there was a feedback why this is the case. P4

Version 3. Shows dependencies in layout. P5

According to the users version 2 supports them best, followed by version 3. Also when comparing the usability problems of version 2 and 3, version 2 supports

best.

28

© itemis AG

Which dialogue was worst?

Answer ID

Version 1 because of the error message. P1

Version 1. P2

Version 1. Error message can be overlooked easily. Message confuses what needs to be done. P3

Version 3. Structure too unclear. P4

Version 1. P5

Version 1 supports worst.

29

© itemis AG

Further improvements

Answer ID• Tooltips should explain the different options in the dialogue (e.g. what does Testing

Support mean). P1

-/- P2

• Hint messages should be bigger. • Nobody wants to read hints. P3

-/- P4

• Description of options as they are not self-descriptive (e.g. when hover them)• An information is necessary that always a basis is generated and that facets can be

added optionally.• Terms are not consistent (sometimes „support“ sometimes „plugin“)

P5

30

© itemis AG

Summary of results

• The results show that version 2 should be realized

• In addition:

• Add tooltips that explain the different options

31

usability test results xtext new project wizard

Design