VisiStat: Visualization-driven,Interactive Statistical Analysis

Krishna SubramanianRWTH Aachen University52062 Aachen, Germanykrishna.prasath.subramanian@rwth-aachen.de

Figure 1: From theselected variables and

properties of thedata, VisiStat infers

appropriatevisualizations and

statistical tests.

userID

layoutDependent

Independent

genderDependent

Independent

speedDependent

Independent

errorsDependent

Independent

ratingDependent

Independent

Participant

Histogram Boxplot Scatterplot Scatterplot-matrix

spee

d

i10 DVORAK QWERTY

15.46

19.64

23.82

28.00

32.18

36.37

40.55

44.73

48.91

53.09

10.86

4.45

?

AssumptionsNormality of distributions

Homogeneity of variances

1-way ANOVAF(2, 57) = 11.82

p < 0.001

0.290 1

2

Pairwise post-hoc test

Tukey's HSD test

Permission to make digital or hard copies of part or all of this work for personalor classroom use is granted without fee provided that copies are not made ordistributed for profit or commercial advantage and that copies bear this noticeand the full citation on the first page. Copyrights for third-party componentsof this work must be honored. For all other uses, contact the Owner/Author.Copyright is held by the owner/author(s).CHI’14 , April 26–May 1, 2014, Toronto, Canada.ACM 978-1-4503-2474-8/14/04.http://dx.doi.org/10.1145/2559206.2579423

AbstractTo promote the practice of sound statistical analysis inHCI, we introduce VisiStat, a tool that allows users toperform statistical analysis by interacting withvisualizations. It guides users to select the appropriatestatistical analysis tasks based on the research questionsthey want to answer. By collocating statistical analysisresults with appropriate visualizations, users are madeaware of data-specific knowledge, which consequentlyimproves their understanding of data and reduces commonstatistical analysis mistakes. In our user study, VisiStathelped users to answer 90% of the research questions theyposed. On average, the users performed four statisticalanalysis tasks beyond their prior experience.

Author KeywordsStatistical analysis; progressive disclosure; interactive datavisualization

ACM Classification KeywordsG3 [Probability and Statistics]: Statistical software; H.5.2.[Graphical user interfaces (GUI)]: .

IntroductionStatistical analysis is an integral part of HCI research.However, there is cause for concern over the lack ofadequate statistical analysis knowledge among HCI

researchers. A survey conducted on research papers inleading HCI journals revealed that 40 of 41 sampledpapers had issues with the reported statistics [1]. Some ofthe prominent issues were inappropriate testing,over-testing, and a disregard for checking assumptions.The immediate effect of this is that it invalidates, or atthe least obfuscates, the author’s findings.

Table 1: Knowledge involved instatistical analysis

Statistical knowledge:

• This includes information suchas selection procedure ofstatistical tests, list ofassumptions for these tests,and interpretation of variousterms in the result.

• This knowledge is a majorbarrier for HCI researchers wholack a strong mathematicalbackground.

• Books, online courses, andseminars attempt to bridge thisgap [3, 4, 6]. However, theserequire major learning efforts.

Data-specific knowledge:

• This includes information suchas shape of the distribution,awareness to outliers, andtransformation of data.

• Data-specific knowledge isupdated during the course ofthe analysis, and datavisualizations often help inrevealing and reminding usersof this knowledge.

Statistical analysis requires both statistical knowledge anddata-specific knowledge to be in the user’s head (Table1). Current statistical analysis software often separatesdata visualization from statistical analysis results (e.g., pvalues or effect sizes). The latter are often presented in anisolated table. This separation between visualization andstatistical results could lead to the analyst overlookingpotential anomalies in the data (e.g., outliers), resulting inmisinterpretations and, subsequently, mistakes withstatistical analysis.

VisiStat attempts to improve the quality of statisticalanalysis in HCI research in two ways: (1) it guides theuser to select the right statistical analysis procedure byembedding the statistical knowledge in the system, and(2) it collocates the statistical results with appropriatevisualizations to remind users of data-specific knowledge.

Related workPrevious research in the field of statistical analysis hasdealt with interactive visualization of raw data andautomated statistical procedure selection.

TouchViz is a visualization tool for tablets that has a rich,interactive design [5]. It uses multi-touch gestures basedon physical metaphors to enable the user to performrudimentary statistical analysis tasks such as clusteringand filtering. However, it is limited to descriptiveanalyses, such as clustering and filtering.

StatWing1 is a web-based, commercial software thatallows users to perform statistical tests. Like VisiStat,StatWing automatically selects the appropriate statisticalprocedure based on research questions and augments theresult with visualizations. However, StatWing displaysgraphs as a result of the analysis. Users may overlookdataset-specific knowledge, apparent in the visualization,because of premature conclusion. The lack of datavisualizations prior to statistical analysis can cause usersto overlook dataset-specific knowledge that may beapparent in the visualization. To prevent this potentialnegligence, in VisiStat, users initiate statistical analysis byinteracting with visualizations.

In the following sections, we present the design principlesemployed in VisiStat followed by a walkthrough of theinteraction and the user study.

Design PrinciplesVisiStat enables users to perform complex statisticalanalyses by placing both statistical knowledge anddata-specific knowledge upfront. This goal is elaboratedin the following design principles:

Embedded statistical procedure decisions: During thepreprocessing stage, users specify the type and role ofeach variable in the dataset. This information is used byVisiStat to select the appropriate visualization andstatistical procedures. This relieves the user from the needfor prior extensive statistical knowledge.

Interactive visualizations: Users initiate statisticalanalysis by interacting with visualizations. This enablesthem to identify potential anomalies in the data beforestatistical analysis. Also, as seen in TouchViz, interactivity

1http://statwing.com

encourages users to perform Exploratory Data Analysis [5].

Number of independent variables

0

interval & normal one-sample t-test

ordinal or interval one-sample mediancategorical (2 categories) binomial test

categorical (3+ categories)Chi-square test

1 with 2 levels (independent groups)

interval & normal2 independent sample t-test

ordinal or intervalWilcoxon-Mann Whitney test

categorical Chi-square test

1 with 2 or more levels (independent groups)

interval & normalone-way ANOVA

ordinal or interval

Kruskal Wallis testcategoricalChi-square test

1 with 2 levels (dependent/matched groups)

interval & normal paired t-test

ordinal or intervalWilcoxon signed-rank test

categorical McNemar test

1 with 2 or more levels (dependent/matched groups)

interval & normalone-way repeated measures ANOVA

ordinal or intervalFriedman test

categoricalrepeated measures logistic regression

2 or more IVs

interval & normal

factorial ANOVAordinal or intervalordered

logistic regression categoricalfactorial logistic regresion

1 interval IV

interval & normalcorrelationsimple linear regression ordinal or intervalnon-parametric correlation categoricalsimple logistic regression

1 or more interval IVs and/or 1 or more categorical IVs

interval & normal

multiple regression

ANCOVA

categoricalmultiple logistic regressiondiscriminant analysis

Figure 2: A decisiontree showing tests in

VisiStat by thenumber of

independent variablesand by the type of

dependent variables

Progressive disclosure: The details of statisticalassumption tests and descriptive statistics are shown ondemand. E.g., an indication of whether each assumptionhas been satisfied or violated is revealed initially. When aviolation occurs ,or upon user request, detailed results ofthe assumption accompanied by additional visualizationsare shown.

Freq

uenc

y

15.46 18.60 21.73 24.87 28.00 31.14 34.28 37.41 40.55 43.68 46.82 49.95 53.09

0

3

60

3

60

3

6

i10

DVORAK

QWERTY

Figure 3: Histogram mode isused to check data for bimodality.

28

14.2

42.1

10.5

Figure 4: Hovering over a meandisplays its numerical value. Theuser selects means forcomparison by clicking on them.

Interaction WalkthroughHere, we describe a typical workflow of VisiStat. Prior toanalysis, users preprocess data to be in long format2

manually or with the help of available preprocessing toolssuch as [2]. Users also specify the column containing UserIDs of participants in the experiment, from which VisiStatinfers the experimental design. For this walkthrough, weuse a dataset from a text entry study that follows abetween-subject design.

2One column per variable; each row is an observation

Inferring visualization based on variable selectionsThe main screen is shown in Fig. 1. Users can select rolesof the variables (dependent or independent) on the left.This aids the user in framing research questions. Also,based on the variable selection, VisiStat selects thevisualizations that are most likely to lead to furtheranalyses. Here, keyboardLayout (categorical independentvariable) and typingSpeed (interval dependent variable)are selected. VisiStat shows a boxplot of the typing speedcorresponding to each layout.

At the bottom of the screen, users can choose otherpossible visualizations for the selected variables to revealdifferent aspects of the data. From the previous example,the user may want to check if the data is multimodal byswitching to histogram mode (Fig. 3).

Interacting with visualizationsElements of the shown graphs, such as bar, box, orwhiskers, are interactive, allowing descriptive statistics tobe revealed on-demand, by hovering with the mousepointer over bands and whiskers of boxplots (Fig. 4).Users also initiate statistical tests through these elements.In this walkthrough, the user has selected the means ofthe distributions to compare them.

Automatically selecting statistical testsVisiStat determines appropriate statistical tests based oncurrent visualization, type of selected variables, and theirroles. Here, the dataset follows a between-subject design,and the user had selected three means of layout, which isa categorical independent variable. VisiStat infers that theuser wants to compare means, and used One-wayANOVA, which is appropriate for the selected variables.Currently, VisiStat implements common tests forinterval-dependent variables (Fig. 2).

Performing necessary assumption testsVisiStat automatically performs necessary tests ofstatistical assumptions. Here, normality of distributionsand homogeneity of variances are tested. Details of thetests, correction procedures (e.g., transformation), oralternative nonparametric tests are disclosed to the userprogressively (Fig. 5).DVORAK

1.00 5.00

Assumptions

Normality of distributions

Homogeneity of variances

Figure 5: Violation of thenormality assumption leads tofurther diagnostic plots.

spee

d

i10 DVORAK QWERTY

15.46

19.64

23.82

28.00

32.18

36.37

40.55

44.73

48.91

53.09

10.86

4.45

outlier

Figure 6: The difference betweenmeans are visualized next to thedata. Hovering over the bracketsshows pairwise differences.

0.290 1

2

0 1

small

medium

large

Figure 7: Effect-size magnitudeand range is visualized in a bar.Hovering over the bar showslegends assisting interpretations.

Viewing resultsBesides the numerical results of statistical analyses, aninteractive layer pertaining to the results augments thevisualization. In Figure 6, the difference betweenconsecutive pairwise means overlays the ANOVA results.Hovering over each pair reveals the difference between thecorresponding means.

VisiStat visualizes some of the numerical results to help ininterpretation. In Figure 7, a color-coded bar is used tovisualize the magnitude and the possible range of η2.

VisiStat overlays the results on the graphs to visuallyremind the user of data-specific knowledge. In Figure 6,outliers can be spotted in the boxplot and be highlightedafter mean comparison tests.

Performing further analysesUsers can perform further analyses by interacting withvisualizations. E.g., the user can select two means in themean comparison results for a pairwise post-hoc test.VisiStat determines an appropriate test based on theselected data. E.g., selecting two normally-distributedmeans invokes a pairwise t-test, with non-parametricalternative of a pairwise wilcox-test being invoked whendistributions are not normal.

User StudyWe conducted a user study to investigate whetherVisiStat aids users to perform statistical analysis tasks andalso to explore VisiStat’s influence on users’ statisticsskills. Our research questions were:

RQ1 To which extent does VisiStat help the users toperform the statistical analysis tasks that they want?

RQ2 To which extent does VisiStat guide the users toperform statistical analysis tasks beyond their priorstatistical knowledge?

RQ3 To which extent does VisiStat enable users to betterspot errors in statistical reports?

These research questions deal with statistical knowledgethat cannot be directly measured. In this study, we usedobservable indicators to represent statistical knowledge aswill be discussed in the Results section.

ProcedureThe structure of our user study is shown in Figure 8.Prior to the user study, our users filled out an onlinequestionnaire, where they self-assessed their expertise onvarious statistical analysis tasks. The lab study consistesof two sessions held on two consecutive days. On the firstday, users performed a pre-test error-spotting task andworked with VisiStat. On the second day, they performeda post-test error-spotting task to assess the difference instatistics skills.

In each error-spotting task, users received metadata aboutone dataset. This included a short description and themotivation of the experiment, the experimental design,and the list of controlled/measured variables and their

data types. User were then asked to read 7 excerpts ofstatistical reports pertaining to the given dataset (Fig 9).They had to identify statistical errors and ambiguousinformation that could jeopardize each excerpt. For eachidentified error, the experimenter ascertained the users’reasoning by asking followup questions. The number oferrors were balanced between the set of excerpts used forpre-test and post-test error-spotting tasks.

Figure 8: Steps in our user studyand the data used to answer eachof our research questions.

Users typed significantly faster with QWERTY layout when compared to DVORAK layout. Unpaired t-test was used t(34) = 3.35 p = 0.9 d = 1.3 indicates large effect size p-value is too high for significance

Figure 9: An excerpt shown inthe error-spotting task. The erroris annotated in red.

For each VisiStat task, users were introduced to adifferent dataset and were asked to note down researchquestions that they were interested in exploring. Then,they used VisiStat to explore the data and answer theresearch questions in a think-aloud session.

We recorded the usage log, screen, and audio. After thesession, we conducted a semi-structured interview forfurther comments about the user experience. Averageduration on the first day was 1h 15mins for each user.

In the post-test error-spotting task, users were tested withthe same dataset as in the pre-test, but with a differentset of excerpts. The post-test was done on the second dayto avoid short-term retention.

ParticipantsEight volunteers (aged between 22-31; 1 female), whowere students and researchers with an HCI background,were recruited from the local university to participate inour study. All participants had knowledge of experimentalstudy design. They rated their statistical expertise fromnovice to intermediate.

ResultsUsers can do their planned analyses with VisiStatFrom the recordings we identified whether users analyzedthe research questions they had planned, and how

confident they were with their analyses. Users identified51 research questions (M = 6 questions/user). Of these,users correctly analyzed and interpreted 46 questions usingVisiStat. There were four questions for which our usershad correct results but were not confident, as VisiStat hadapplied a parametric test despite the violation of anassumption (because there were no alternative tests in thissituation). This suggests that the presence of assumptionindicators made the users be aware of this violation and,thereby, be more cautious with their interpretation. Onaverage, users came up with 6 research questions and wereable to answer 90% of them confidently using VisiStat.

VisiStat enables users to perform statistical tests be-yond their previous knowledgeFor the formulated research questions, users needed to doon average 7.5 statistical tests beyond their self-reportedknowledge. Of these, VisiStat guided them to perform onaverage 4.25 statistical tests. In an extreme case, User 7,who self-rated statistical knowledge very low, performedmost tests beyond previous knowledge (Fig. 10). However,his comment indicates a dangerous potential to beoverdependent on VisiStat: “... I don’t have to worryabout the different tests and just let the system choosethe appropriate test”. Nevertheless, User 3 remarked thatVisiStat improves his conviction on performing analysis:“VisiStat reinforces the belief that what I have picked isthe right test, which gives me confidence ...”.

Influence on error-spotting skills is inconclusiveUsers’ utterances while identifying errors were annotatedand classified into various types based on the statisticalterms (e.g., p-value, test statistics, and the significancetest used) and the type of error (e.g., incorrect value andincorrect interpretation). We collected both thetrue-positives (correctly identified errors) and the

false-positives (identified errors that did not exist). Thedifferences in the quantity and quality of error descriptionindicates the knowledge that participants acquired frominteracting with VisiStat. Overall, the number of

!34

Does VisiStat guide users to do tasks they haven’t done before?

Use

r ID

1

2

3

4

5

6

7

8

Score on Survey

0 15 30 45 60

Use

r ID

1

2

3

4

5

6

7

8

Number of tasks done beyond knowledge

0 2 4 6 8

Figure 10: Users’ scores in theonline questionnaire and thenumber of tasks completedbeyond prior knowledge.

4

2

0

-2

-4

4

2

0

-2

-4

score on pre-test

score on post-test

Figure 11: Change in scores forall users between pre- andpost-test error-spotting.

true-positives increased from 17 to 24, while falsepositives decreased from 17 to 11. The score in theerror-spotting task is calculated by subtracting thenumber of false positives from the number of truepositives. On comparison between the pre- and post-tests,the score increased for five users, remained constant forone user, and decreased for two users (Fig. 11).

LimitationsWe had a limited number of users for the study and theywere not equally motivated. This influenced the amountof time they spent working with the systems (from 15 minto 1.5 hr) and, subsequently, their performance in the userstudy. In the error-spotting task, behavior changed frompre-test to post-test. Some users were more adventurousand tried to find more errors, whereas some users weremore cautious and exercised restraint. As a result, thelack of uniformity in user involvement affected their score.These confounding factors contribute to the lack ofconsistent patterns of how VisiStat influenced theirbehavior.

Future WorkWe plan to investigate how VisiStat complements othermodes of learning, e.g., video lectures, self-learningmaterials [6], and interactive statistics tutorials3. We alsoplan to include more statistical analysis procedures inVisiStat, e.g., statistical tests for categorical dependentvariables.

3E.g., http://swirlstats.com/

ConclusionWe presented VisiStat, an interactive data analysisapproach that infers visualization and statistical testsbased on users’ interactive input. Our user study showsthat VisiStat (1) allows users to find answers to theirresearch questions, and (2) assists them to performstatistical analysis tasks beyond their baseline statisticalknowledge. We envision that VisiStat will improve thequality of statistical analysis in HCI community.

VisiStat is open-source. An online demo and source codeare available at http://hci.rwth-aachen.de/visistat.

AcknowledgementsThis work was funded in part by the German B-ITFoundation.

References[1] P. Cairns. HCI... Not As It Should Be: Inferential

Statistics in HCI Research. In BCS-HCI ’07, 195–201.[2] S. Kandel, A. Paepcke, J. Hellerstein, and J. Heer.

Wrangler: Interactive Visual Specification of DataTransformation Scripts. In CHI ’11, 3363–3372.

[3] J. Lazar, J. H. Feng, and H. Hochheiser. ResearchMethods in Human-Computer Interaction. WileyPublishing, 2010.

[4] H. C. Purchase. Experimental human-computerinteraction: a practical guide with visual examples.Cambridge University Press, 2012.

[5] J. M. Rzeszotarski and A. Kittur. TouchViz:(Multi)Touching Multivariate Data. In CHI EA ’13,1779–1784.

[6] J. O. Wobbrock. Practical statistics forhuman-computer interaction. In Annual Workshop ofthe HCI Consortium (HCIC ’11).