BIG.DATA.The Problem and the Solution to 21st Century Organizational Innovation.Trever PearsonPA 740Professor Hyde12.10.12

What is BIG DATA?

DEFINITION. WHY IS IT PROBLEMATIC?

Big Data is a phenomenon defined by the rapid acceleration of the expanding volume of high velocity, complex, and diverse types of data which require advanced technologies and methods to enable their collection, storage, dissemination, management, and analysis.

Increased velocity of available data is faster than most organizations can keep pace with.

Data synthesis requires advanced technologies and appropriate staff and expertise on an ongoing basis.

Implementation requires structural and organizational culture change.

… And failure to respond will leave a lagging organization seriously behind.

TechAmerica Foundation (2012)

Origins & Trends.

Data Storage.

Global Data Storage has increased from 0 to over 300 exabytes between1986 and 2007.1

The type of global data stored has changed from 99% Analog in 1986 to 96% Digital in 2007

1986 1993 2000 20070

50

100

150

200

250

300

350

Data Storage in Exabytes

1986 1993 2000 20070%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Detail: % Exabytes

DigitalAnalog

1 (5 Exabytes = 10^18 gigs: Enough to contain every word ever spoken by all humans on Earth. MGI (2011)

Origins & Trends.(cont…)

Computational Capacity.

Computation capacity has grown from 0 to over 300 exabytes of traffic from 1986 to 2007.

Information-producing devices such as, mobile phones, tablets, sensors etc… have doubled since 2000. Coupled with personal computing, traffic in these areas increased from under 40 to nearly 90% of all data created form 1986 to 2007.

1986 1993 2000 20070

50

100

150

200

250

300

350

Computation Capacity(Million Instructions per

Second

1986 1993 2000 20070%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Detail: % Million Instructions per Second

Personal ComputersVideo Game ConsolesMobile Phones/PDAServers and MinframesSupercomputersPocket Calculators

MGI (2011); Economist (2012)

Origins & Trends. (cont…)

The storage required for all of this data doubled between 1999 and 2002, a 25% compound annual growth rate.

1.8 zetabytes of data (the amount of 200 billion 2-hour HD movies) were created globally in 2011; an amount projected to double every year.

800 exabytes were created in 2009, projected to increase 44 times by 2020.

It’s just like the universe, increasingly and exponentially expanding.

MGI (2011)

What is All this Data?DATA TYPES. DATA SOURCES. 15% Structured

(database or spreadsheet data)

85% Unstructured (email, video, blogs, call center conversations, Facebook posts, Tweets, etc…)

Customer transactions with personal information and consumer behavior like Visa, Amazon, etc…)

Multimedia content such as High-Res health procedure videos, YouTube, etc…

Social Media such as Facebook and Twitter

Sensors and devices used in industries such as, retail, healthcare & automotive

Economist (2012)

Why is BIG DATA Important for Management? The effective response to Big Data is crucial for leading

organizations to outperform their peers.

Companies are projected to increase operating margins by

more than 60% with the effective response to BIG DATA.

Management decision making will be built upon evidence

and information.

Data driven decisions are just plain better decisions.

McAffee & Brynjolfsson (2012)

“You don’t manage what you don’t measure”.

Why Go BIG?

HOW WILL BIG DATA HELP? WHO WILL BIG DATA HELP?

By… Replacing human decision-

making with automated formulas where appropriate

Reducing inefficiencies Creating transparency Discovering variability Reducing security threats and

crime Increasing ability to predict

mission outcomes Reducing or eliminating waste …just being innovative.

The five sectors to gain the most from the use of Big Data:

Health Care Public Sector

Administration Manufacturing Retail Business/Organization

using Personal Local data

MGI (2011)

Stakeholders.

WHO IS AFFECTED? HOW ARE THEY AFFECTED?

The Public

Policy Makers

Contractors

Employees

Government transparency, Bureaucratic efficiency…

…Privacy Informed decision-making,

evidence based legislation Monitoring contract

deliverables, reporting Facilitation in workplace tasks,

enhanced communication, etc…

Increased transparency over organizational activity

BIG DATA and Management.OPPORTUNITIES. CHALLENGES.

Data-driven organizations perform better on measures of financial and operational results than those who do not

Data facilitate efficient processes, saving time and money

Data lead to innovation Data will ultimately lead to

funding.

Data-driven decision making and collection processes require organizational cultural change

Strong Leadership is necessary to set clear goals and to ask the right questions

Skillful and talented Data/IT Specialists must be on staff.

Lack of statistical and technical skills in the labor force

Potential cost of implementation

McAffee & Brynjolfsson (2012)

How Does BIG DATA Work?Step 1. Source Data: Speed, Type and Amount.

What kind and how much data are we working with? Assessing how hard it is to access Determining how it needs to be transformed Identifying the technologies to facilitate the process

Step 2. Data Preparation: Cleansing and Verification. What do the data need for operational requirements?

Define methods required for data prep such as: Standardization, verification, filtering, etc…

Step 3. Data Transformation.What is required to leverage the data? Unstructured data may be broken down and presented in a structured

format Data sources can be aggregated to determine not-so-obvious

relationships between data types

TechAmerica Foundation (2012)

How Does BIG DATA Work? (cont…)

Step 4. Business Intelligence/Decision Support.Tools, methods, techniques to leverage data Data Mining Visualization/Simulations Keyword Searches & Syntax Analysis

Step 5. Analysts/Visualization.How should the data be used? Present data visually so it can be explored Use data as is to support/enhance/improve existing

organizational processes

TechAmerica Foundation (2012)

How Does Big Data Work? (cont…)

TechAmerica Foundation (2012)

What Does BIG DATA Need? Staffing

Infrastructure

Funding

Performance objectives

related to desired mission

outcomes

A Data-Driven

organizational culture

Openness to organizational

change

Data Analysts/IT Specialists, etc…

Data storage, Software, Hardware, Connectivity, etc… Technological investment

Standards/Metrics to compare operational efficacy with mission outcomes

Data Prioritization as driving force of organizational direction and the culture to support it.

Data prioritization will require change!

How to Implement BIG DATA?

1. Identify and Define mission objectives that need Big Data solutions2. Assess current organizational capability, data sources, and technical

requirements3. Identify success criteria, implementation timeline, potential

subsequent phases, required staffing levels, and “entry point”1. Streams as entry point for high-velocity data needs2. Un-bounded database/warehouse infrastructure for high-volume

data needs3. “Hadoop”1 or similar type technologies for high-variety data needs

4. Execute the plan as required5. Review on an ongoing basis

1 The Apache Hadoop project develops open-source software for reliable, scalable, distributed computing. TechAmerica Foundation (2012), Apache Hadoop

How Do You Know BIG DATA is Working? Assessment of mission outcome achievement

with improvement measures including increased savings, improved efficiency, etc…

Identification of gaps in the links of the process chain, if any (see slide 13)

Assessment of decisions being made (are the right data available to facilitate the process?)

What are your Data/IT staff telling you?

Advice for Government.• Expand and invest in the talent pool by creating a formal track for IT/Data managers with training and certification in BIG DATA Analysis and technologies.

•Establish and broaden coalitions between industry academic and associations to develop professional standards and shared best practices for the field.

•Expand “college-to-government service” internship programs focused on technical aspects of BIG DATA.

•Strengthen and expand Office of Science and Technology Policy to facilitate further research into new techniques and their applications to important problems across program and policy sectors.

•Align incentives to promote data sharing for the common good.

•Provide further guidance with industry and stakeholders on privacy and data protection practices.

•Develop intellectual property policies to promote innovation.

•Support necessary underlying IT/Communications infrastructure

MGI (2012), TechAmerica Foundation (2012)

Sources of Resistance.Political resistance to BIG DATA may be minimal, resulting from a history of activity including:Government (Library of Congress, Bureau of Information Resource Management)Finance (Banks, Credit Card companies)Internet search engines (Google)

…HOWEVER…Bottom-up Resistance is likely

The Public

Employees

Contractors

Privacy concerns and the notion of “Big Brother”

Data errors and the documentation of mistakes

Less room for error, increased competition and accountability

FAQs.1. How do you know if you have a big BIG DATA problem?

2. How do you obtain insight from your data?

3. Which technology is right for my organization?

4. How long should it take to implement?

5. What skills/expertise are required on staff?

6. What about Privacy?

1. When available data is beyond your ability to manage or when tapping into the insight it provides is problematic.

2. Start by placing mission objectives at the heart of every decision. While this might require change, even the more traditional change management practices may be of service. Let your Data staff tell you what they need.

3. It depends on your mission objectives and the type/amount/speed of data you need to inform your decisions. To start, build upon what you already have.

4. Start with small, manageable steps and allow for constant evaluation and revision. If the first phase takes longer than 6 months, you’re too slow.

5. Data Analysis and Communication, Technical skills, Database Management, and good ol’ fashioned Critical Thinking.

6. As with any data collection/sharing advancement, policies must be adjusted to address issues of privacy as they affect the organization within the context of the standards set in place (statutory or otherwise). Congress is working on it. As far as the public is concerned: Welcome to the 21st Century.City A.M. (2012)

References.

Identifying what big data means to you. (2012, Feb 24). City A.M. London.

McAffee, A., Brynjolfsson, E. (2012). Big Data: The Management Revolution. Harvard Business Review. Pp. 59-68.

Manyika, J., Chui, M., Brown, B., Bughin, J., Dobbs, R., Roxburgh, C. Byers, A.H. (2011). Big Data: The Next Frontier for Innovation, Competition and Productivity. McKinsey Global Institute (MGI).

TechAmerica Foundation (2012). Demystifying Big Data: A Practical Guide to Transforming the Business of Government. Washington, D.C.

Geography matters as much as ever despite digital revolution, says Patrick Lane. The Economist. (2012).

Trever Pearson.

Trever Pearson is a third-year Master’s student in Public Administration at San Francisco State University. With an emphasis in Policy Analysis

and Finance, his interests lie mostly in evidence-based improvement in the policy arena in sectors such as health, education,

finance, and income security.

Trever comes from a solid background in health care policy implementation and evaluation in

the San Francisco public health network. He is currently working as a Data Analyst for Curry Senior Center, a community clinic serving the

elderly in San Francisco’s

Tenderloin neighborhood. His achievements there include the development of agency-wide data collection and reporting processes for service quality improvements and contract reporting.

With coursework in Urban Administration, Financial Management and Applied Statistics, Trever aspires to use BIG DATA and research solutions for the improvement of state and federal policies and agency operations.