2

Quote

Success is getting what you want. Happiness is wanting what you get.

Dale Carnegie

http://www.quotationspage.com/

3

Who are we?

Matt Sharpe (Document Manager)

Peter Centgraf (Technical Lead)

Lisa Edelman (Client Liaison)

Lorrianne Nault (Project Manager)

Adrian Tang (User Testing Lead)

4

What is CitiStat?

• CitiStat Goals:– Ensure that City departments have the

resources and infrastructure to deliver services

– Provide operating savings

– Set Benchmarks for comparison to industry standards

– Focus resources on priority initiatives, such as Pittsburgh Clean Neighborhoods

• CitiStat is a management data reporting process

– Provides data to Mayor and committee for review

– Charts, graphs, and maps are used for data analysis

– Added importance due to 5-year budget plan, Act 47 compliance

Mechanical Repairs - Downtime > 7 DaysOcotber 2001 - September 2003

-

20

40

60

80

100

120

140

10/1

5-10

/28

11/1

2-11

/25

12/1

0-12

/31

1-14

/1-2

7

2-11

/2-2

4

3/11

-3/25

04/0

8-04

/21

05/0

6-05

/19

06/0

3-06

/16

07/0

1-07

/14

07/2

9-08

/11

08/2

6-09

/8

09/2

3-10

/06

10/2

1-11

/04

11/1

7-12

/01

12/1

6-12

/29

01/1

3 - 01

/26

2-10

/2-2

3

3-10

/3-2

3

04/0

7-04

/20

05/0

5-05

/18

06/0

2-06

/15

06/3

0-07

/13

07/2

8-08

/10

Nu

mb

er o

f V

ehic

les

Trendline

5

How do we fit in?

• Goal: To create a system that supports visualization and analysis of performance data, promotes efficient data entry, and enables more effective use of city resources.

• The MHCI challenge– Create a fully-functional project that can be put into practice, even if in limited form

– Give the City’s technology group something they can support, but not have to implement

• Special Concerns– Ease of learning

– Ease of use and efficiency

– Sustainability of technology

– Scalability

– Delivery of fully-functional prototype

Raw DataManage data,

where it comes from

Dynamic Reporting

Visualization of mgt needs

Departments Management

A B

Final ReportingPresentation of value-add data

Public / Regulators

C

6

What process are we supporting?

7

Where did we begin?

• Determined a scope of work– Focused on Department of Public Works (DPW), Bureau of Operations

• Investigated CitiStat process– Worked backwards along flow of information

– Office of Management and Budget (OMB) to DPW

– Contextual Inquiry and Interview techniques

• Analyzed the most effective ways to improve it– Constructed diagrams to describe

and confirm our understanding

– Recorded key observations and their links to CitiStat goals

8

How does the CitiStat process work?

WorkDepartment of Public Works

Enter Data

Foreman System

Copy DataExcel template

worksheets

Update Copy to History

File

Alert Tell OMB Staff

History complete

PrintLook at data set, search for trends

AnalyzeChanges,

problems, useful findings

GraphCreate graphs of

relevant data

ComposeBriefing books to

share data

ReportCitiStat meetings

ImproveAssist DPW with resources, etc

9

What challenges did we find?

• Redundancy (6)– Data points: CI1-14, CI1-2, CI2-12, CI9-5, CI8-1, CI4-14

• Shared Understanding (4)– Data points: CI6-1, CI1-1, CI6-2, CI7-25

• Effort (5)– Data points: CI1-13, CI9-2, CI8-2, CI9-6, CI4-14

• Coordination (7)– Data points: CI7-12, CI4-24, CI4-28, CI4-18, CI9-3, CI4-19, CI7-21

• Accuracy (4)– Data points: CI9-7, CI6-13, CI6-8, CI7-17

• Also identified main themes in the data and other key observations

Breakdowns

10

How are we changing it?

• Input interface allows DPW employees to input the data directly into live database

• Immediate access to the data by both DPW and OMB Analysts

• Export Excel report, data tables, and graphs

• Ability to quickly change the order of metrics, add new metrics, and remove metrics no longer needed

• Comments and/or description for metrics

WorkDepartment of Public Works

Enter DataForeman System &

Input Interface

AnalyzeQuick graphs

and charts

CreateBriefing books to share data

ReportCitiStat

meetings

WorkDepartment of Public Works

Enter Data

Foreman System

Copy DataExcel template

worksheets

Update Copy to History

File

Alert Tell OMB Staff

History complete

PrintLook at data set, search for trends

AnalyzeChanges,

problems, useful findings

GraphCreate graphs of

relevant data

ComposeBriefing books to

share data

ReportCitiStat meetings

ImproveAssist DPW with resources, etc

11

How does it support CitiStat?

• Office of Management and Budget (OMB)– Eliminated manual conversion to different forms and formats

– Visual data analysis based on automatic graphs and data tables

– Flexibility to explore different cross-sections of data by year or division

– Documentation for the meaning behind the numbers

• Department of Public Works (DPW)– Some data could be imported from current tracking system

– Ability to view data to prepare for CitiStat meetings

– Can use data to inform internal decisions

12

How does it work?

• Input interface allows DPW to input CitiStat data directly into the database

• Change interface provides flexibility for the order of metrics and allows a user to add additional or remove pre-existing metrics

• Analysis interface allows OMB and DPW staff to view overall trends and analyze the data

Access Database

InputInterface

AnalysisInterface

ChangeInterface

ForemanSystem

13

How does the change interface work?

• Add or delete categories and metrics

• Specify metric description, type, labeling style, and unit of measure

• Manipulate the ‘metric tree’ as needed

Metric Tree

Search

Add new items

Name, Description

and type

Units

Labels

Links

14

How does the input interface work?

• Find section to input in metric tree

• Input data according to month and budget vs. actual data

• Correct previous data as needed

Metric Tree

Attributes

Value

History

Navigation

Search

15

How does the visualization interface work?

16

How does the metric tree work?

• View graphs and data tables through navigation of the data metric tree

• Search for metrics

• Description of the metric provided (as entered in the change interface)

17

How does the graph area work?

• View or Hide the data table

• Copy data table or graph and paste in Word, Excel, etc

• Adjust the X-axis and Y-axis

• Customize the graph title

18

How do the graph option controls work?

• Graph Data: Actual Value, Planned Value, % of Plan, and % Change by Month and Year

• Graph Type: Line or Bar Graph

• Compare by Years or Divisions

• Exponentially expands the number of possible graphs and comparisons

19

How do the summary tables work?

• When a category is selected, a summary data table is shown

• Can copy and pastesummary table

20

What other features does it support?

• Export data table or graphs

• Print graphs

• Load / Save graphs as favorites

• Export Excel report (comparable to current template)

21

What can be done in the future?

• Graphs– Add trend-line to bar charts

– Add a pie chart or other graph types

– More flexibility on editing the graph and changing the display

• Communication features– Add contact information in metric description

– Provide functionality to email metric contacts about discrepancies or misunderstandings

– Add notes section for specific metrics

• General Improvements– User-test and improve input and change interface

– Contextual help

– Save screen real-estate by using collapsible pallets

22

Demo

23

How did we get here?

Development

ResearchContextual

InquiryDesign

Paper Prototypes

Heuristic Evaluation

User Testing

Documentation & Deployment

Develop

Test

Design

24

How did we uncover context?

• Learned more about city management– Investigated Baltimore, MD CitiStat program

– Interviewed public works department of Wheeling, WV

– Purchased CompStat Paradigm by Vincent Henry

• Familiarized ourselves further with Pittsburgh– On-going review of Pittsburgh Post-Gazette

• Explored more design-centered visualization techniques– Examined Enterprise Dashboard technology, persistent,

on-going, interactive data displays

– Samples from the gaming world

– Data Visualization instructors from HCII at CMU

Development

ResearchContextual

InquiryDesign

Paper Prototypes

Heuristic Evaluation

User Testing

Documentation & Deployment

25

How did we discover and define problems?

• Observation of Pittsburgh’s CitiStat program– Exposure and understanding of Pittsburgh’s

culture

– Hands-on experience with how the system currently worked

• Contextual Inquiries and Interviews– Contextual Inquiry: An approach to answering

the question, “What should we build to help CitiStat employees do their work even better?”

– Gained a better understanding of tasks and data flow

– Areas for improvement in current processes

– Environmental and technology constraints

– Having conversations with users in the context of their work creates richer data

Development

ResearchContextual

InquiryDesign

Paper Prototypes

Heuristic Evaluation

User Testing

Documentation & Deployment

26

What did we do with that data?

Development

ResearchContextual

InquiryDesign

Paper Prototypes

Heuristic Evaluation

User Testing

Documentation & Deployment

Strength Not StrengthAverage

Category Speed Flexibility Simplicity

Tabs

Mac

Excel +

Pivot

Tree

27

How did we evaluate our designs?

• Heuristic Evaluation: Small team of evaluators to test an interface based on recognized usability principles including

1. Visibility of system status 6. Recognition vs. recall

2. Match between system 7. Flexibility and efficiency of use and the real world

3. User control and freedom 8. Aesthetic and minimalist design

4. Consistency and standards 9. Error recovery

5. Error prevention 10. Help and documentation

• Think-Aloud Usability Testing: Asks users to “think aloud” as they work– Discover specific problems with the interface from actual DPW or OMB

users

– Understand the way the users think about the problem in order to present information appropriately

Development

ResearchContextual

InquiryDesign

Paper Prototypes

Heuristic Evaluation

User Testing

Documentation & Deployment

28

How did we utilize User Testing?

Testing Goals

Scenarios

Results Sheet and UARs

Consolidation

Solution Brainstorming

Prototype Iterations

Develop

Test

Design

29

What did it take to get here?

• 12 contextual inquiries / interviews

• 29 user tests

• 10,782 lines of code

• 461 presentation slides

• 363 emails

• 11 computer failures (Matt)

• 2 lost or injured USB drives

• 500+ cups of coffee

• 1,960 crackers

30

What are some future directions?

• Useful improvements– Automatic unit conversions

• System Integration– Pull data directly from other data sources

• History– Login functionality to keep track of individual’s changes

• Apply this tool for the other departments within Pittsburgh

31

Who has supported us along the way?

• We would like to thank:– Joanne Foerster

– Pittsburgh Department of Public Works

– Pittsburgh City Information Systems

– Pittsburgh Office of Management and Budget

– Bob Kraut

– Carolyn Rose

– Bonnie John

– MHCI class, especially our critique team

– And all of you for attending our presentation

32