wcre - oct 2007 - 1 temporal model explorer - farah and lethbridge temporal exploration of software...

WCRE - Oct 2007 - 1Temporal Model Explorer - Farah and Lethbridge

Temporal Exploration of Software Models

HannaFarah

IBM Ottawa

TimothyLethbridgeUniversity of Ottawa


Temporal Model Exploring (TME)Key Ideas

Smoothly slide a document back and forth as it appeared in time

Mark spots of interest (snapshots) to jump back and forth to

Filter the edit events– E.g. filter out position changes in a diagram so items

stay in their final position

Annotate history easily


Hypotheses

Participants using TME– Can solve modeling problems

• more quickly• more accurately

– Prefer it to not using TME

Participants using final positioning filtering in TME– Can solve modeling problems

• more quickly• more accurately

– Prefer it to not using final positioning


Study

Participants– Twelve IBM employees

• Managers• Software developers• Coop students

Experiment steps– Participants were first given a consent form– Given a short training session in the TME feature – Given three class diagrams and asked questions

(more details of experiment blocking) (more details of questions asked)


Measured performance results

No statistically significant difference in speed or accuracy

– When using TME or not– When using original position

– (more details of the measured results)


Questionnaire results 1

I found that the TME feature helped me understand class diagrams more quickly

– 4.2 / 5

Using the page-up and page-down keys, I found that a useful set of steps in the development of the model (snapshots) were presented

– 4.1 / 5

I would use the TME feature if it was available to me in my work environment and I was asked to understand a class diagram.

– 3.9 / 5


Questionnaire results 2

TME feature resulted in me taking a longer time to answer the questions

– 2.4 / 5

TME feature was awkward to use– 1.7 / 5


Future work 1

Similar study with larger models that have multiple diagrams

Giving the TME tool to a group of users– use it in their daily work– surveying them to get more feedback

Additional filters– i.e. filter on name changes


Future work 2

Automatically detecting snapshots based on:– Context– Time

Deploying the TME feature in word processors and other tools


Summary

TME: Temporal exploration using a slider– Final positioning filtering– Adding, removing, or editing at ANY time of

• snapshots (controlled granularity)• temporal annotations

– Simple, would not add to ‘feature overload’

Study– Users gave positive feedback

A patent has been filed on key aspects of this work


Questions?


Supplemental slides


Experiment blocking (back)

Blocking by– Problem– Problem sequence– Person creating original model– Model– Participant ability– Treatment pattern

• Order of No TME TME with final position TME with original position


Experiment blocking grids (back)

Table 1: Allocation of participants to models and treatments

T1: Only final

diagram

T2: TMEP original

pos

T3: TMEP

final pos Investments-

A A1, E3 G1, K2 C2, I2 Investments-

B B1, F3 H1, J2 D2, L2 Elections-A C1, H3 F2, I3 A3, J3 Elections-B D1, G3 E2, L1 B3, K1

Airline

I1, K3, J1, L3

A2, C3, B2, D3

E1, G2, F1, H2

Table 2: Allocation of participants, controlling for experience

Participant Experience Treatment

pattern Problem sequence

A Higher 1 t23 IAE B Lower 1 t23 IAE C Higher 1 t32 EIA D Lower 1 t32 EIA E Higher t32 1 AEI F Lower t32 1 AEI G Higher t23 1 IAE H Lower t23 1 IAE I 1 t32 AIE J 1 t23 AIE K t23 1 EIA L t32 1 EIA


Sample question asked (back)

Imagine the airline regularly flies from Ottawa to Toronto, but wants to create an extra charter flight tomorrow from Ottawa to Toronto. To enable this, the system will have to create instances of one or more classes. Which classes will the system have to make instances of? No need to worry about booking any passengers or assigning crew.

Figure 2: The Airline System class diagram used in the evaluation of TME


More details of the measured data (back)

Table 3: Measures of speed and accuracy. All values are normalized, where the average for all participants over all models =1. A value greater than 1 means faster or more accurate.

Hypothesis and measure

Mean Max Min Std. Dev

95% Conf

Experts’ Mean

H1a: Time taken by participants in treatment T1 (without TME)

0.99 1.45 0.57 0.24 0.14 Same

Time taken by participants in T1 and T2 (TME)

1.00 1.21 0.77 0.12 0.07 Same

Conclusion: No significant difference H1b: Accuracy of participants in treatment T1 (without TME)

1.00 1.21 0.74 0.15 0.08 1.05

Accuracy of participants in T1 and T2 (TME)

1.00 1.13 0.90 0.07 0.04 0.98

Conclusion: No significant difference in general, but With experts showing a slight and not-statistically- significant loss of accuracy with TME

H2a: Time taken by participants in T2 (original positions)

1.04 1.51 0.58 0.26 0.14 Same

Time taken by participants in T3 (final position filter)

0.97 1.21 0.76 0.14 0.08 Same

Conclusion: Minor but not-statistically-significant Improvement in speed

H2b: Accuracy of participants in treatment T2 (original pos.)

1.01 1.26 0.80 0.14 0.08 1.00

Accuracy of participants in treatment T3 (final position)

0.98 1.08 0.79 0.09 0.05 0.98

Conclusion: No significant difference

wcre - oct 2007 - 1 temporal model explorer - farah and lethbridge temporal exploration of software...

Documents