the key to big data modeling: collaboration

© 2015 Universal Data Models, LLC - All Rights Reserved – Not to be copied or distributed without permissions 1

Aug 26th, 2015 Webinar,

By Len Silverston, Universal Data Models, LLCSponsored by Embarcadero Technologies


Purpose

Share Keys

to Big Data Modeling and How to Collaborate


Agenda

• Big Data Overview

• Data Modeling in Big Data

• Collaboration Principles


Big Data

Big data is a broad term for data sets so large or

complex that traditional data

processing applications are inadequate. Wikipedia

3Vs – Volume, Velocity, VarietyBy 2020 - 44 zettabytes!

Mostly unstructured


Unstructured DataInformation that either does not have a pre-defined data model or is not organized in a pre-defined manner. Wikipedia

How can data have

no structure?

Is "unstructured" data

merely unmodeled?*

* Structure, Models and Meaning’ Seth Grimes, Information Week,


New Landscape - NoSQLKEY VALUE DATABASES

GRAPHDATABASES

DOCUMENT STORES

MongoDBMUMPS DatabaseObjectDatabase++OrientDBPostgreSQLQizxRethinkDBRocket U2SednaSimpleDB

Solr

TokuMXOpenLinkVirtuoso

OpenLinkVirtuosoOracle Spatial and GraphOracle NoSQL DatabaseOrientDBOQGRAPHProfium SenseR2DFROISSemblent Lionsgatesones GraphDBSPARQLCitySqrrl EnterpriseStardogTeradata AsterTitanTripleBitVelocityGraphVertexDBVivaceGraphWeaverWhiteDBOhmDB

RedisXAPKV - solid-state drive or rotating disk[edit]AerospikeBigTableCDBClusterpoint Database ServerCouchbase ServerFairCom c-treeACEGT.MHibariKeyspaceLevelDBLMDBMemcacheDB (using Berkeley DB or LMDB)MongoDBNoSQLzCoherenceOracle NoSQL DatabaseOpenLink VirtuosoTarantoolTokyo CabinetTuple space

KV - eventually consistentApache CassandraDynamo

Oracle NoSQL DatabaseProject VoldemortRiakOpenLink Virtuoso

KV – orderedBerkeley DBFairCom c-treeACE/c-treeRTGFoundationDBHyperDexIBM Informix C-ISAMInfinityDBLMDBMemcacheDBNDBMKV - RAM[edit]AerospikeCoherenceHazelcastmemcachedOpenLink Virtuoso

BaseX

CloudantClusterpoint DatabaseCouchbase ServerCouchDBCrateIODocumentDBElasticsearcheXist

HyperDexInformixJackrabbitLotus Notes (IBM Lotus Domino)MarkLogic

AllegroGraphArangoDB

BlazegraphBitsy

BrightstarDB

Cayley

DEX/Sparksee[2]

Filament

GraphBase

Graphd

Graph Engine[3]Grapholytic

Horton

HyperGraphDB

IBM System G Native Store

InfiniteGraph

InfoGrid

jCoreDB Graph

Neo4j

OntotextGraphDB

Orly


In this age of Big Data,is there ‘less of a need’ or

‘more of a need’ for data modeling?(or ‘no need’ or the ‘same need’)


Why Model?

8

DATA

Understand

Design?

Common semantics?


UNDERSTANDOVERSTAND

What are customers saying about our products?

What exactly do we mean by a customer?Is a prospect that has signed a contract but not paid yet, a customer?Is a person that only bought from us over 10 years ago a customer?Is an organization that bought a minor item from us a customer?Is sales volume based on orders, invoices, payments, or GL posts?

What are we predicting our sales volume to be this quarter?


REQUIRESTEXTCON

TEXT


Traditional

MODEL

(and DESIGN)LOAD

EXPLORE/

QUERY

DATA EXPLORE

‘Schema on write’


Big Data

LOAD QUERY MODEL

NoSQL STORE

EXPLORE

But Fast and Agile!

‘Schema on read’


Data modeling in Big Data

Customer---

NoSQL DATABASE

Documents---

Product---

Key values---

Conceptual/business data model

Understanding

Logical/physical data model

Architecture/Design

RELATIONAL DATABASE

(i.e., Data warehouse/data mart)

May transfer into structured database (using models)


Big Data Modeling Considerations

• Changes nature of modeling– Later– Modeling for understanding

• Design considerations - performance and scalable• Changes where physical structures reside: in code• Shifting functions to programming–Performance– Security– Integrity

• Lately, SQL interfaces over NoSQL


When to Model First,When To Explore First

Explore First

When format cannot be predicted in advance (Rapidly changing data structures)

When you need to keep ‘data as is’

Continually new sources of data

Don’t know if valuable (exploratory)

Huge amounts of information (e.g. streaming terabytes per minute)

E.g. Cyber terrorism, Sentiment Analysis

Model First

More predictable data structure

When there is some flexibility to modify/conform data

Stable and known sources

Know that it’s valuable

Reasonable amount of information for relational

E.g. Customer demographics, Product info, Sales History


What Does ‘Agile’ Mean?• Customer solution – deliver value• Flexible• Fast• Iterative• Sustainable – constant

pace• Quality design

(and efficient)

• Human Factors– Communication - face to face– Collaborative– Trust– Motivation– Ongoing reflecting and adjusting

Quotes from principles behind the Agile Manifesto can be found at http://agilemanifesto.org/principles.html

“Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.”

“Welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage”

“Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale”

“Business people and developers must work together daily throughout the project.”

“Working software is the primary measure of progress.”

“Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.”

“Continuous attention to technical excellence and good design enhances agility.”

“The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.”

“Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.”

“Simplicity--the art of maximizing the amount of work not done--is essential.”

“At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.”

“"The best architectures, requirements, and designs emerge from self-organizing teams.”

Can we do this in data modeling?


What is Agile - What is NOT Agile?• Agile data modeling

– Deliver flexible, quality data models that facilitate sustainability and deliver value, in a quick, iterative, collaborative way, building trust and motivation

• NOT agile data modeling

– Quick & dirty

– Excuse to develop more silos

– Without quality or understanding

PERSON ORGANIZATION

PARTY

PARTY ROLE

PARTY RELATIONSHIP

PARTY CONTACT MECHANISM

FACILITY

CASE

WORK EFFORT

ROLE TYPEWORK EFFORT ROLE

CONTACT MECHANISM

SUPPLIERCUSTOMER WORKER PARTNER

CONTACT MECHANISM TYPE

PROJECTPROGRAM TASK

FIXED ASSET

ASSIGNMENT

WORK EFFORT

ASSOCATION

COMMUNICATION EVENT

FIXED ASSET

PRODUCT

GOOD SERVICE


How can we perform agile data modeling?

1. Re-use2. Quick broad brush data

model3. Correct model for

correct purpose4. Prioritize5. Deliver6. Understand motivations7. Have lots of choices

available

See Article: “Data Modeling’s Role in Agile Development”http://tdwi.org/articles/2010/07/07/data-modeling-agile-development.aspx

RE-USE!

http://tdwi.org/articles/2010/07/07/data-modeling-agile-development.aspx


EPISODE TYPE

EPISODE TYPE ID

DESCRIPTION

HEALTH CARE EPISODE

HEALTH CARE EPISODE ID

PATIENT PARTY ID (FK)

PATIENT ROLE TYPE ID (FK)

INCIDENT ID (FK)

EPISODE TYPE ID (FK)

EPISODE CREATE DATE

HEALTH CARE VISIT

HEALTH CARE VISIT ID

PATIENT PARTY ID (FK)

PATIENT ROLE TYPE ID (FK)

CONTACT MECHANISM ID (FK)

FACILITY ID (FK)

FROM DATE

THRU DATEINCIDENT

INCIDENT ID

INCIDENT TYPE ID (FK)

INCIDENT DATE

DESCRIPTION

EMPL RELATED IND

INCIDENT TYPEINCIDENT TYPE ID

DESCRIPTION

PATIENT

PARTY ID (FK)

ROLE TYPE ID (FK)

SYMPTOM

SYMPTOM ID

HEALTH CARE EPISODE ID (FK)

SYMPTOM TYPE ID (FK)

DESCRIPTION

SYMPTOM TYPE

SYMPTOM TYPE ID

DESCRIPTION

VISIT REASON

VISIT REASON ID

HEALTH CARE VISIT ID (FK)

SYMPTOM ID (FK)


DESCRIPTION

HEALTH CARE DELIVERY

HEALTH CARE DELIVERY ID

HEALTH CARE VISIT ID (FK)


HEALTH CARE OFFERING ID (FK)

FROM DATE

THRU DATE

DELIVERY NOTES

Re-use to understand

Universal Data Models


We must understand the

data and therefore

continue to develop data

models, even in this 'Big

Data' era.

Come on! Get into

the new mindset of

today’s Big Data! We

need to do things

differently today!

How can you use the data without

first understanding it?

Also, it’s important that we all

use common semantics.

Are you trying to slow

us down and continue

to try to enforce

bureaucracy?!

Data Modeler Data Scientist


Biggest Issue:

“Mine”

Data ‘Mine’ing


Key to Big Data Modeling:Data ‘Ours’ing


Keys To Collaboration

• Shared Purpose

• Understand Motivations

• Develop Trust

• Listen

• Manage Conflict


Vision?

Mission?

1. Shared Purpose


Can you state the exact mission statement of your organization?

(without looking it up first!)

Be Honest

BE HONEST


Position Versus Interest

INTERESTS A

POSITION A

INTERESTS B

POSITION B


2. Understand Motivations


Motivational Model - Sponsorship Map


What Does The Business Really Need?

• Insight?

• Buying behavior?

• Assessment?

• Prescriptions?

• Predictions?


Who is the MOST important person to know their motivations?

A. The Most Influential Sponsor?

B. Your Boss?

C. Your Most Difficult Person Who Is the Greatest Obstacle in Your Effort?

D. Yourself?


Core Elements of TrustKeys to Trust3. Trust


• Character–Integrity

–Intent

– Vulnerability/openness

• Competence–Capabilities

–ResultsFrom “The Speed Of Trust” By Stephen M. R. Covey

Keys to Trust


To Listen, ACCEPT

(A) ware, attention, alert

(C) are

(C) onfirm, check

(E) mpathize

(P) urpose

(T) otally (with all senses)

L

4. Listen and ACCEPT


Keys to Trust5. Conflict Management


What is the first thing to do in a conflict?

A. Define your strategy for winning?

B. Understand their perspective?

C. Don’t react?

D. Figure out a win-win?

E. Something else?


Don’t React

EventFeelings/Thoughts Emotional

PhysicalStories

Reaction

FreezeFlightFight

Mess

Step 1. Don’t react


Respond

EventFeelings/Thoughts

Emotional,Physical,Stories

Data

Stop – observe.Data?Questions?

Response

Intelligent Actions

Step 1. Don’t react


Big Data Modeling

Model To Understand – Even if after viewing data

Collaboration is the key

Find Common Purpose

Understand Motivations

Develop Trust

Listen

Conflict Management


What Will You Do With This?


Questions or More Info?

[email protected]

Twitter: @lensilverston

For info on template Models:www.embarcadero.com/products/er-studio-universal-data-models

http://www.universaldatamodels.com/

mailto:[email protected]


Creative Commons Image Attributions

Much thanks to those who provided the creative common images in this presentation. Thanks for:

Stars https://www.flickr.com/photos/tom_hall_nz/17317951241/sizes/sq/ All rights reserved

by Kiwi Tom

Sky https://www.flickr.com/photos/cubagallery/9679210392 © All rights reserved

by ►CubaGallery

License

https://farm5.staticflickr.com/4038/4323076259_64348b9b69_b.jpg

https://www.flickr.com/photos/tom_hall_nz/

https://www.flickr.com/photos/cubagallery/9679210392

https://www.flickr.com/photos/cubagallery/

https://creativecommons.org/licenses/by/2.0/

the key to big data modeling: collaboration

Software