1

. . . . . . . . . Revision date – 7 November 2012 – Copyright UMass Lowell

Weave Information Visualization for Social Responsibility

Figure 1 – Obesity in the United States. CDC county-level data shows the percentage of the adult population that is

characterized as obese. Red indicates the highest levels, blue the lowest.

Institute for Visualization and Perception Research

University of Massachusetts Lowell

Computer Science Department

Lowell MA 01854

2

. . . . . . . . .

Table of Contents

1. Vision ..................................................................................................................................... 3

2. Social Responsibility ....................................................................................................... 6

3. First Open Source Product: Weave ............................................................................. 7

4. Data ........................................................................................................................................ 8

5. Analysis ................................................................................................................................ 8

6. Visualization .................................................................................................................... 10

7. Session State .................................................................................................................... 12

8. User Experience ............................................................................................................. 14

9. The Near Future: High-Impact Activities ............................................................... 16

9.1. SimpleWeave ............................................................................................................... 16

9.2. STEM (Science, Technology, Engineering and Mathematics) Learning .. 16

9.3. Large Group Collaboration ..................................................................................... 16

9.4. Data Commons ............................................................................................................ 17

9.5. Sharing Visualizations .............................................................................................. 17

9.6. InfoMaps ........................................................................................................................ 17

9.7. ADA-compliant Weave .............................................................................................. 18

10. Summary: Looking to the Future ........................................................................... 19

3

. . . . . . . . .

1. Vision

Our mission is to change the world by democratizing data -- by providing access to

data anywhere by anyone for any purpose.

By equalizing the ability of organizations and individuals to use data, we all will be

better able to make informed decisions and to influence the social, political,

economic and environmental distribution of resources.

Challenges to Democratizing Data:

Access to data is limited. Our world faces significant problems, challenges that

cannot be resolved without access to data. Yet we keep information locked away

from the very people, organizations and institutions that can best help address

those problems.

Data that is accessible is not easy to understand. Most data is simply not

available in clear, usable formats.

It is expensive to access and analyze data. Most data remains unintelligible

except to a privileged population – those with the resources and tools to

manipulate, analyze and interpret it.

We will approach a true data democracy when everyone – regardless of position or power

in society – is able to understand, explore and interpret data in depth; to discover,

generate and test hypotheses; and to make informed decisions.

To make the best use of today’s veritable explosion of data, we need methods and tools

that allow every individual, organization and institution to raise their awareness of and

increase access to data. We also need ways to make analysis and sharing of that data

easier. Such tools will affect large-scale social change by determining trends, causes,

correlations and structures in that data.

Several specific steps must be taken to attain a true data democracy.

We must improve access to data. The first step is to provide universal web-

based access to all available public data. To achieve this, we need software tools

that facilitate secure distribution and access.

We must provide education and training so that anyone can analyze and use

data for decision-making. True lasting change cannot occur without knowledge

and understanding. Data is the foundation of such knowledge.

Data Information Knowledge

4

. . . . . . . . .

Data becomes information once it is structured. Information becomes knowledge

once it is analyzed and visualized. That step – analysis and visualization – allows

knowledge to inform decisions. A well-informed public holds decision makers

accountable and enables all to make better decisions.

We must provide free, modern, web-based tools and training to all. To

become truly informed, the public must have access to data analysis tools that are

free, up to date, powerful, flexible well-maintained and easy to learn and use.

These tools must provide universal web-based access to all available public data

and must provide methods to share discovered knowledge.

To fulfill our mission, our goal is to provide technology that will make it easy for

everyone to access, analyze and share data.

Though not a short-term goal, technology for democratizing data is a realizable one.

In late 2011, the IVPR, in collaboration with the Open Indicators Consortium,

released Weave Version 1.0 (see iWeave.org) which enables access to public

data.

Weave Version 2 (anticipated release: Spring 2013) will include SimpleWeave, a

streamlined version, for users and small organizations with little or no IT support.

SimpleWeave will allow users to easily upload, present and share their data.

Weave Version 2 will also include document retrieval (Infomaps) which will

automatically link data to associated documents, reducing the time and effort for

separate online searches. Weave Version 2 will support collaboration, real time

sharing of data by up to hundreds of remote users.

Weave Version 3 will include an internet data engine that will enable search and

access of any and all publicly available data much like a Google search for

documents today.

The final anticipated version, Weave Version 4, will include access to large

national and international data sets.

These are exciting possibilities and we’re on our way.

Weave will empower small and large organizations, especially non-profits, to access,

analyze, visualize, and use data, and affect change through the power of data and

knowledge.

Data Questions + Citizen Analyses + Visualizations => Insights and Informed Decisions

5

. . . . . . . . .

The IVPR, its faculty, staff and students, are committed to providing the public with

free tools to access, visualize, and analyze data.

Here’s how you can help us support our goal.

Financial

Fund collaborative research or applied projects

Fund one of the high-impact activities (See Section 9)

Donate to the University of Massachusetts Lowell Foundation -- Weave Project

Actions

Download Weave and use it to analyze data or make data available

Work with us to make national, international, and global data sets available

Encourage the widespread use of Weave

Promote Weave, blog about it, post about it, download it, and bring it up in

conversation whenever possible.

Promote the concept that data exists as a resource for all

Figure 2 – Showing trees in Boston MA (each is labeled and probeable).

6

. . . . . . . . .

2. Social Responsibility

The Institute for Visualization and Perception Research (IVPR) will continue to develop

free and open-source software specifically designed to support not-for-profit

organizations and the general public in the exploration and presentation of data.

We are using the latest software technologies to develop software tools that support all

aspects of data visualization and analysis. These tools can also help disseminate web-

based visualizations.

Free open-source software provides four freedoms or rights (as described by the Free

Software Foundation):

The freedom to run the program for any purpose.

The freedom to study how the program works and to change it to make it do what

you wish.

The freedom to redistribute copies so you can help your neighbor.

The freedom to improve the program and release your improvements (and

modified versions in general) to the public so that the whole community benefits.

Figure 3 - CDC Obesity data visualized by state

7

. . . . . . . . .

3. First Open Source Product: Weave

Weave is an open-source state-of-the-art Web-based Analysis and Visualization

Environment which provides software tools that researchers, educators, analysts, trainers,

students and the general public can use to analyze and visualize remote, local or

distributed data (iWeave.org).

Weave is the IVPR’s fifth-generation visualization system and incorporates 20 years of

research with embedded patented algorithms. The previous generations were desktop

versions designed and used to solve complex problems in a variety of application areas

including drug discovery, medicine, economics and national security.

Weave is being developed in conjunction with the Open Indicators Consortium (OIC) and

is currently in use by the Massachusetts Department of Early Education and Care, the

Massachusetts Department of Higher Education and many organizations and government

agencies including those in Boston, Chicago, Columbus, Grand Rapids, Kansas City,

Seattle, Arizona, Connecticut and Rhode Island.

People are using Weave to solve complex problems and to further the goal of data

democratization. (See iWeave.org for examples.)

Figure 4 - MBTA bus routes in Boston, Massachusetts. Red points indicate bus stops.

8

. . . . . . . . .

4. Data

Providing universal secure web-based access to all available public data involves:

1. Raising public awareness that relevant data exists

2. Finding the data.

3. Using accepted access standards.

We’ve addressed all three.

We use standard query systems to access data, whether that data is in a local

spreadsheet, a database on a server or distributed databases.

We are developing the National Data Commons (NDC) to make available very large

databases that are of interest to the public and organizations.

We are developing SimpleWeave (see section 9) to facilitate Weave installations and

further broaden data access. SimpleWeave will have numerous

visualizationexamples and Wizards to demystify the visualization process. It will

enable the average person to become data literate and make informed decisions, and

produce and share their own visualizations.

We and the OIC are are committed to the concept of open data and making publicly

available data accessible to all.

Figure 5 -- Weave-based website of OIC member, Connecticut Data Collaborative

5. Analysis

To gain knowledge, raw data must first be converted to information. This is

accomplished by curating the data which involves examining the data for errors or

9

. . . . . . . . .

omissions and adding metadata (data about the data such as information about its source,

collection and its structure). Computational tools can be used to efficiently handle

missing data, to identify correlations, outliers and patterns in data, to generate and

validate hypotheses and to convert these into actionable decisions.

Weave provides significant support for data analysis—from clustering to trend analysis.

Weave has both server-based computational engines such as the R-project for statistics or

analysis and client-based computation. An equation editor allows a user to define any

function on the data and generate new computed columns, titles or labeling.

R-project can perform various analyses including exploration of trends or identification

of clusters of data. The resulting clusters or ordering of data or attributes can be used as

controls for other high-dimensional visualizations or highlighted as selections in any

visualization. Missing data can be imputed, correlations computed (as per Pearson and

others) and multiple clusterings compared and all visualized. Weave also includes other

advanced analytic tools including Principal Component Analysis, Support Vector

Machines and Clustering, Bayesian Networks, Multidimensional Scaling and Association

Rules.

In addition, classifiers can be generated, tested and validated.

Once results have been identified (classes, trends, outliers, messages to convey), the

analyst can use Weave tools to print high quality visualizations, to generate interactive

visualizations on the web, and shortly, to make these web-pages ADA compliant.

10

. . . . . . . . .

6. Visualization

Visualizations are critical in the decision-making process. We can easily determine the

largest value within a column, but to determine its distribution is very difficult without

visualization. Visualizations support decision makers as well as the discovery process.

Figure 5 - Lowell Foreclosure data, Lowell, Massachusetts, showing census tracts and individual lots

Visualization Tools

All Weave visualizations can be embedded in web pages or used within Weave’s

integrated visualization and analysis system. The visualizations are rich, broad and

flexible. They range from exploratory and provide interactive experiences that are also

aesthetically controllable.

Analysis

Weave tightly couples analysis and visualization thereby improving the efficiency and

effectiveness of analysis. The user can generate a self-organizing map, a multi-

dimensional scaling visualization or any other analysis-integrated visualization.

11

. . . . . . . . .

Advanced Visual Analytics

We have developed new powerful visualization and analysis technologies not just for

simple data but also for high-dimensional data (hundreds of thousands of variables and

millions of records). We have validated these high-dimensional visualizations in many

application areas including drug discovery, health records monitoring, economics and

national security. One of these tools, RadViz, has been extended to a multiple clustering

visualization called vectorized RadViz. Vectorized RadViz shows stable sub-clusters

within data and is a remarkably powerful visual and analytic tool. We have also enhanced

RadViz dramatically for Weave to provide strong user support.

Graphs and Networks

With Weave, graphs can be visualized and then linked with other visualizations to greatly

enhance visual analytics. In order to provide stable views of graphs, we have defined

anchor points (user defined, context defined or computed from some clustering or MDS-

like algorithm). The anchor pointlocations stay the same throughout the algorithm

iterations thereby extending and speeding up current layout algorithms.

Merging Text and Visualizations

We have made our patented InfoMaps available within Weave. This provides an

integration of Weave with document collections, whether locally available, in databases

or in content management systems. For example, when selecting a subset within a

visualization, relevant documents in a collection can be identified and highlighted.

12

. . . . . . . . .

7. Session State

The Session State architecture on which Weave is based provides a mechanism that

tracks each activity performed within Weave. This mechanism can be used to address two

issues central to modern data analysis: the need to continually evolve new analysis tools

and the ability to validate and replicate computational results.

1. To continually improve, expand and update data analysis tools, is it imperative

that we understand how researchers and data explorers analyze data, discover

patterns, and gain insights.

2. To validate computational or visual results, analysts must be able to replicate

discoveries or analyses.

Weave tracks every user action and produces a session history. These single-user or

multiple-user session histories can be visualized and interpreted, used to build user

profiles, to construct a recommendation system, and to support training and maintenance.

Usage patterns may be identified and studied, sessions compared, user groups discovered

and anomalous users identified. Sessions can be shared and used as templates. Thus, any

user can view the visualization and the process including all the analytic steps used to

generate it. A session history becomes an object which can be distributed for others to

see and explore.

13

. . . . . . . . .

Figure 6 - Nitrogen load in New England streams displayed using river basin shape files. Snapshot

would be stored in the session state

14

. . . . . . . . .

8. User Experience

Commercial Grade Software

Usable, robust, maintainable and reliable software is expected from any software vendor.

The IVPR has developed commercial-grade software for the past 20 years, including

graphics software for Lockheed and Intel, risk assessment software for the Avon Breast

Center at Massachusetts General Hospital and surgical simulations with haptics tactile

feedback system for Sensable Technologies.

Figure 7 – The Rhode Island Data Hub website is based on Weave visualizations. Other live Weave-based websites

include: Metropolitan Area Planning Council (Boston), Michigan Data Collaborative (Grand Rapids) and Mid-Ohio

Regional Planning Commission.

15

. . . . . . . . .

Agile Software Development Process

Weave was developed using the Agile Software Development Process with continual

feedback from Weave users and developers. The current base of users in the OIC

provides a strong usability evaluation group. Internally our students and faculty

continually test all new capabilities in academic classes, in research projects and in

response to OIC members. Bugs are reported and quickly addressed.

Learning Community

The OIC supports a learning community that shares deployment experiences, supports

each other via group emails and holds an annual user group conference. A feature list is

maintained and open to the public for suggesting long-range future desired features and

also serves to identify usability issues. Weave is being extended to be ADA compliant

and large multiple user collaboration interfaces are being designed and implemented.

Future Capabilities

Weave has proven to be a welcome and timely system. Its future capabilities discussed in

the next session are broad and our design focus continues to focus on providing a high–

performance and usable system for everyone to analyze and visualize data.

Figure 8 - Lowell, Massachusetts – data displayed at parcel level

16

. . . . . . . . .

9. The Near Future: High-Impact Activities

Future releases of Weave will include several high-impact tools thatwill support our goal

of providing access to data anywhere by anyone.

9.1. SimpleWeave

As mentioned above, we are developing SimpleWeave, a release specifically designed for

the average person with no IT background. SimpleWeave will not require use of a server.

A small not-for-profit community without an IT department will be able to easily install

Weave. The user will download the SimpleWeave file, execute it, answer a few questions

such as what maps they would like to use (county, street, neighborhood, ...), where their

data resides (Excel file, database), and select a page from a template.

SimpleWeave will read the appropriate shape files from the UMass Lowell/OIC server,

set up the hosting pages and open a number of preset visualizations computed from the

user’s data. A wizard will then lead the user through the selection of visualizations and

interactions. This simple set-up solves a persistent problem for individuals and small non-

profits that do not have or cannot afford IT expertise, thus addressing a major

impediment to data democratization.

9.2. STEM (Science, Technology, Engineering and Mathematics) Learning

Weave session history tracks the actions taken while using Weave. We can support the

analysis and classification of Weave session data. This information can be used to study

how students approach learning, particularly in the STEM fields. Students could be asked

to use Weave to complete a STEM-focused task (for example, “study the migration of

birds”). As each student proceeds in his/her own way the system collects the student’s

session history. By examining these session histories, teachers can observe the individual

student’s problem-solving approach, monitor progress and determine where a student

may have gone astray or may have used an unconventional but effective strategy. Since

session histories can be clustered and analyzed, patterns may appear which show

common approaches to that exercise (patterns of steps, and even more interesting, a

classification into a very small number). These can be analyzed with demographic and

other data to identify correlations.

Such STEM tools will extend the classroom beyond current physical boundaries. They

also will allow researchers to explore learning theories and can be used to provide STEM

experience in non-STEM areas.

9.3. Large Group Collaboration

Weave will support groups of users collaborating across the world. Weave's current

collaboration feature allows a student who is at home to participate in class by following

17

. . . . . . . . .

what the teacher is doing and using Weave to interact while the teacher and class watch.

By extending collaboration to hundreds of users, Weave will improve user experience in

a variety of scenarios from large-scale distributed classrooms to groups that wish to

explore complex problems with experts from many different disciplines or viewpoints.

Such tools dramatically enhance webinars by providing users with a shared interactive

system for education and group discovery.

9.4. Data Commons

We will provide new and open access to the growing body of national and international

data (data sets and databases) now available on the Internet from both government and

non-government sources. Each data set contains metadata, information that describes that

data, in order to facilitate access and comparison. A user will be able to search for

databases that contain, for example, the query " availability of “Meals-on-Wheels” in

cities with low crime and fewer than 100,000 people” " The Data Commons will search

its databases as well as the public indexed databases and return links to those databases

that satisfy the query. That data will then be viewable in Weave.

Such a tool will allow any organization to compare itself with others, to see how others

are evolving, to perform queries across the collection of data and to enrich their research

with these external datasets.

9.5. Sharing Visualizations

We will provide tools to exchange and share visualization sessions. Weave's session

history allows a user to save both a visualization and the process (the sequence of steps

taken by the user) that generated that visualization. A user will be able to send the

visualization session (session history) to a colleague who can examine not just the

visualized data but also the steps that generated it. By making the session public, others

can see the steps that were taken, the results that were obtained and can also begin to

understand and improve the process. This is very similar to the STEM application of

Weave described above.

This tool will allow sites and organizations to permit their members or clients to upload

and share visualizations and sessions. The session history can be replayed at any time

and can be used as a teaching or training tool.

9.6. InfoMaps

As mentioned above, future versions of Weave will include InfoMaps technology to help

users visually manage and monitor information from multiple sources. InfoMaps allows

the user to pre-select subject areas or sources to monitor. This could include patents,

publications, newsletters, pages, or other text collections, whether private, public or

18

. . . . . . . . .

subscribed. Once these selections are made, InfoMaps automatically updates and presents

the information in a user-defined visual format on the desktop.

InfoMaps’ technology has been integrated with Weave. Any user data selection in Weave

automatically filters documents on the InfoMaps display for those most relevant.

InfoMaps links Weave-selected data and provides immediate access to relevant

documents, news and web site information in real time all within a single display. For

example, a user may select a subset of a scatterplot that might include population, obesity

and education. Relevant documents or news reports would then be made available within

a window using an InfoMaps layout. This addresses the problem of dealing with the

massive quantity of information available and needed by individuals by making that

information accessible automatically as the data is explored. Without InfoMaps, users

must continually perform web search queries and cut and paste between the Weave and

Information Retrieval displays and pages. InfoMaps is easy to configure and use. It takes

only five minutes to set up and one minute to learn.

9.7. ADA-compliant Weave

According to the Americans with Disabilities Act of 1990 (ADA), any information made

available online by a publicly-funded entity must also be made available to people with

disabilities. This poses a challenge to these agencies that wish to use visualizations to

share information, since visually impaired individuals may not have access to the

information contained in a visualization in the same way that others have. The main goal

of visualization is to take data, most often amounts that are too large to easily grasp by

reading a table or text, and communicate a message at a glance. The blind or visually

impaired do not have the benefit of gaining a message at a visual glance. The challenge is

to make that message accessible and convey it to visually-impaired users so that they

comprehend the same message as a person with no visual impairments. Another

challenge is to create an automated process that detects the messages contained within a

visualization.

Current methods to make visualization accessible to visually impaired individuals utilize

“tags”, descriptors that are read to the user via commonly used screen-reading devices

such as Window Eyes or Jaws. These tags must be manually created for each

visualization, resulting in a scalability issue. To make visualizations accessible to the

visually impaired, we propose to use the session history feature of Weave, which would

allow us to share specific descriptions of each visualization. This method would use

Weave session history to automatically detect certain features of the visualization and

provide that information to the user via a screen-reading device.

As described above, the Weave architecture is based on the concept of session states. All

significant actions made within the system, including the visual parameter settings, tool

properties and user interactions are recorded and this “snapshot of history” is stored as a

session state. The session state contains all of the information required to restore a given

Weave visualization or instance. To create a visualization that is accessible to the

19

. . . . . . . . .

visually impaired, the data contained within the session state can be automatically passed

to a screen-reading device to provide an audible description of the visualization to the

user.

Although the Weave session state contains all of the necessary information to

automatically re-create a visualization, we must determine what part of this information

is needed to accurately describe the visualization. We will develop a model of the syntax

that is required to accurately describe a visualization in English. Once the model is

designed it will be tested on both sighted and visually impaired individuals. Then, with

an understanding of the descriptors that are needed to communicate the message of a

visualization, we will apply the model to Weave using the session state data to present the

visualization data in spoken English. This new ADA-compliant visualization will then be

tested on both sighted and visually impaired individuals.

Although the primary focus of this project is to address ADA compliance needs, the

ability to provide audible information about visualizations is also applicable to the

general population, particularly those users whose eyesight is compromised due to age or

from a short-term illness or injury.

Figure 10 – Demographics of Atlanta, Georgia metropolitan area by county

10. Summary: Looking to the Future

Looking to the future, the problems we face with data require new approaches, methods

and tools that will allow every individual, group and organization to easily access,

analyze and share data.

20

. . . . . . . . .

Weave provides these tools. Weave is free, powerful, modern, flexible, well-supported

and easy to learn and use. Weave provides universal web-based access to available public

data. This allows the public to make informed decisions and thus to affect change.

Our mission is to democratize data. We will change the world by making data available

through Weave — providing visualizations for everyone — thereby enabling informed

decision-making.

Weave and its future releases will bridge the current data divide. Having data access is a start. Having tools that

support and encourage continued access and analysis will ensure progress toward our goal of equal access to

data. That is Weave -- technology that will enable everyone, organizations and individuals, to make informed

decisions to influence the social, political, economic and environmental distribution of resources.