authority and data dr. scott renner [email protected] 26 august 2008 draft for discussion only for...

Authority and Data

Dr. Scott [email protected]

26 August 2008

DRAFTFor discussion only

For Limited External Release

© 2008 The MITRE Corporation. All rights reserved

Authority and Data

What could the term “authoritative data” mean in an enterprise like the DoD?

There are four distinct meanings in use today– Agreed data vocabulary– Appropriate source system– Authorized data producer– Approved data for decision-making

All four concepts are necessary in a very large enterprise

To avoid confusion, we need uniform terms of reference– Each concept clearly defined– Distinct terms for distinct concepts

This briefing presents those concepts and proposes terms

2


3

Preliminary Terms of Reference

Producers Consumersinformation


4


Consumers

information

dataProducers


5


desktop computers

handheld devices

weapon systems

sensor devices

Information is producedby people operating…


6


application

application

application

application

sharedinformation

space

…with some sort of applicationposting information into

a shared information space


7


application

application

application

application

sharedinformation

space

application

application

application

application

… pulled or pushed to other applicationsfor the benefit of other people

operating various things


8

operated bymission

organizations

built by acquisition

organizations


application

application

application

sharedinformation

space

application

application

application

operated bymission

organizations

built by acquisition

organizations

controlled by mission

organizations

DataProducers /Information

Owners

DataSources

ConsumingApplications

Consumers /Operators /

Users

Infospace


9


application application

DataProducers /Information

Owners

DataSources

ConsumingApplications

Consumers /Operators /

Users

Infospace

Common Vocabulary:All things describing semantics and syntax

Data models, elements, taxonomies, ontologies, …


Key Characteristics of the Enterprise

No effective central authority– Over producers, consumers, builders, and funding

Instead, we have an “enterprise of enterprises”– Common purpose– Limited autonomy

Often no single sovereign over all participants

10

What could the term “authoritative data” mean in an enterprise like the DoD?


What Is The Enterprise?

11

China SyriaFranceGermany

UN

State TreasuryEPAJustice

US

NavyArmy

DoD

AFMC AFSOCAMCAETC ACC

Air Force

AustraliaUKCommercial NGOs

This enterprise is formed from several sub-enterprises, all having limited autonomy, and some purpose in common

View in slideshow

mode


Key Characteristics of the Enterprise

No effective central authority– Over producers, consumers, builders, and funding– Several enterprises, common purpose, limited autonomy– No single sovereign over all participants

Very large enterprise– Many people, large conceptual domain

Some processes and information needs are not stable– Processes vary from highly-structured to ad hoc– Flexibility is often essential

Information needs range from highly structured datato completely unstructured data

12

What could “authoritative data” mean in an enterprise like this?


Authoritative = Agreed Data Vocabulary

Information / data sharing won’t happenunless people understand the data in the same way– Establish a semantic match– Cope with representation mismatch

When we write down this common understanding,we call it a “common vocabulary”

Agreement doesn’t have to be voluntary for everyone– Some people can create the agreement– Others, subject to their authority, can be told to adopt it– (There are limits on how far this will work)

13

When organizations agree on semantics and representation, we call the result an authoritative data vocabulary

vocabulary


Example

Each COI develops its authoritative data vocabulary– Glossaries– Taxonomies– Data models and data elements– Schemas– Ontologies– Whatever they need to record, use, and teach

their common understanding of data

COI members are expected to understand and use it

14


A data source is an application (aka a system, or Program of Record) that provides data via services

Usually this data isstored internally

Authoritative = Appropriate Data Source

15

application

dataservice interface



A data source is an application that provides data via services

Other applications copy this data for local processing– Today, via web service (XML over HTTP)– Yesterday, in comma-separated files over FTP

16

3

2

5

4

61


9


The fateful day– Instead of getting data from the original source… – An application obtains data from a copy– Or a copy of a copy

17

8

1

35

4

67

2


17

1615

14

13

12

11

10


Over time, the result is chaos– Copies of copies, latency, error “corrections”– Applications give different answers for same question– Nobody knows which applications are still needed

18

89

1

3

2

5

4

67



19

ADS

When the appropriate source system is established,we call it the authoritative data source


Controlled Redundancy Can Be OK

By the way, redundancy isn’t always bad– Controlled redundancy can be useful– If you can keep the sources consistent

20

ADS

ADS

ADS


Each COI has its authoritative data vocabulary

The DoD Metadata Registry (MDR) is the authoritative data source for these vocabularies

Federated registries provide redundant, consistent access

Example

21

FederatedRegistry

FederatedRegistry

MDR

design time

runtime


Authoritative = Authorized Data Producer

Data producers are mission organizationsthat create and maintain data as part of their assigned responsibilities– Systems are not producers– Systems are acquired for

and operated by producers– Individual human operators are not producers– Individuals are part of a producer organization

22



Data producers are mission organizationsthat create and maintain data as part of their assigned responsibilities

Data producers are accountable tothe information owners

Information owners have rights/duties for data– What data will be collected– What level of quality to maintain– Who is allowed to access or update– How these rights are delegated/assigned– This is a mission role, not acquisition

Commander (not program manager) has authority

Delegating and assigning these rights creates an information owner hierarchy

23



Data producers are mission organizations that create and maintain data as part of their assigned responsibilities

Data producers are accountable to the information owners

Information owners have rights/duties for data,exercised within an information owner hierarchy

Many producers can be authorized/assigned to create and maintain the same kind of data– Each JFACC is authorized to create its own ATO

A single data source may support several data producers– Every JFACC uses the same application to create ATOs

24

When a mission organization has the right & responsibility to produce some kind of data,

we call it an authoritative data producer


Example

Each COI has its authoritative data vocabulary

The DoD Metadata Registry (MDR) is the authoritative data source for these vocabularies

Each COI is the authoritative data producerfor its own vocabulary

MDR

COI #1 COI #1


Authoritative = Approved For Decisions

Consumers use data to make decisions

Commanders responsible for those decisions have authority to determine the source of that data– They don’t let just anybody provide the data– Rules and procedures specify information needs,

sources & producers– Commander as infospace authority establishes the rules

Different commands may have different rules/procedures– Sources and producers approved by one

may not be approved by another

26

When a mission organization directs their deciders to use data from a particular source and producer, we call that the

approved data, approved source, and approved producer


Example: Air Mobility Command (AMC)

AMC obtains certain aeronautical navigation data from the National Geospatial Agency (NGA)– NGA is the authoritative data producer– Some NGA system is the authoritative data source

AMC provides corrections of this data to NGA– NGA incorporates these, over time; however… – AMC also maintains its own corrected data– AMC is the authoritative producer of these corrections– AMC’s Table Management Distribution System (TMDS) is

the authoritative source for this corrected data

AMC commander directs AMC consumers to use TMDS data– TMDS data is the approved data for AMC consumers– TMDS is the approved source

27Source: Tina Woodbury (AMC), Neil Custer (CSC) – March 2005


Summary

Authoritative data vocabulary– Semantics and representation established

by agreement among “important” organizations

Authoritative data source– Application designated to provide specified data

Authoritative data producer– Organization with right & responsibility

to create and maintain specified data

Approved data source and producer– Source/producer directed for use within an organization

28

Four concepts of “authoritative data”, four distinct terms


OK, So What?

Let’s see how these concepts and terms help describeCPM Data Governance

CPM data governance begins with a data problem– Some consumer needs certain data, and can’t get it– Portfolio stakeholders cooperate to solve the problem

Two ways to distinguish these data problems– Does the needed data exist?– Is the solution a matter for system developers only?

This results in four categories of solution

29


30

Kinds of Data Problems and Solutions

Net-enable the existing systems

Build or change systems to collect the missing data

Make data sharing agreements

Change TTP

No

Users andOperators

Yes

OnlyAcquirers

Who makesthe change?

Does the needed data exist?

HARDER


31





Change TTP

No

Users andOperators

Yes

OnlyAcquirers



Infospace authority validates the data “need”, advocates for required change, and directs use of the approved data sources that are created


32





Change TTP

No

Users andOperators

Yes

OnlyAcquirers



Acquirers net-enable and/or modify the authoritative sources


33





Change TTP

No

Users andOperators

Yes

OnlyAcquirers



Authoritative data producers change their activities to begin producing the needed data


34





Change TTP

No

Users andOperators

Yes

OnlyAcquirers



Authoritative producers make sharing agreements with infospace authority


Inferences

People sometimes ask: Who owns the data element?Usually not a good question; however… – It belongs to the COI with authority over the vocabulary– Not to a data producer

Many producers can create that kind of data– Not to a data source

Several sources may share that kind of data

Data producer to data source is a many-many relationship

There can sometimes be several sources and producersfor a given kind of information– A single source is desirable, not always feasible– Often desirable to have many producers

Consumers may sometimes choose among several authoritative sources and/or producers

35


Conclusion

All four concepts are necessary in a very large enterprise

Without distinct terms, confusion will ensue

Within the data/info management community,this is our own “shared vocabulary” problem– Doesn’t really matter if we use the proposed terms– Does matter if we understand the concepts

and have distinct terms for each

36


References

Net-Centric Information Management (2005)http://handle.dtic.mil/100.2/ADA464442

DoD Directive 8320.2, Data Sharing in a Net-Centric Department of Defensehttp://www.dtic.mil/whs/directives/corres/pdf/832002p.pdf

DoD 8320.2-G, Guidance for Implementing Net-Centric Data Sharing (2006)http://www.dtic.mil/whs/directives/corres/html/832002g.htm

My Two Cats Are a Community of Interest (2006)http://dodccrp.org/events/2006_CCRTS/html/papers/157.pdfhttp://www.dodccrp.org/events/2006_CCRTS/html/presentations/157.pdf

Net-Centric Information Sharing (2005)

A Style Guide for Common Vocabularies (2005)https://wwwd.my.af.mil/afknprod/DocView.asp?DocID=701841

37


38

BACKUP


39

Applications Post & Pull Through Services

Data Source Data Consumer

post service pull service

sharedinformation

space(physical)

This is how most people envisioned the

shared space in the early days of NCDS;

however…

controller


40

Applications Post & Pull Through Services

service request/response

Data Source Data Consumer

sharedinformation

space(virtual)

People now often envision the infospace as a virtual

construct, which exists as the collected policy decisions of

the infospace authority

controller

authority and data dr. scott renner [email protected] 26 august 2008 draft for discussion only for...

Documents

terms data

mitre corporation

defineddistinct terms

automated applications

data visible

rights reserved6those

consuming applications

things applications