university of namur faculté d'informatique precise research center - database engineering...
TRANSCRIPT
University of NamurFaculté d'informatique
PReCISE Research Center - Database Engineering Groupwww.info.fundp.ac.be/libd
PReCISE
- A (sort of) spatio-temporal view of DB reverse engineering -
Jean-Luc Hainaut
February 5, 2014 Stevens Award lecture WCRE-CSMR 2014
Data matters most
but where has all the semantics gone?
2
• Introduction
• Understanding data semantics
• Data models
• Tracing data semantics
• Recovering hidden data semantic
• Is data semantics recovery that important, actually?
• Summary and conclusions
3
Introduction
4
1. To study the concept of data semantics in business applications
2. To identify and evaluate the techniques used to represent data semantics
3. To observe how these techniques have evolved in time and in different cultures.
4. To discuss the methods used to recover the semantics lost when poor representation techniques have been used.
Objectives of the lecture
5
1. The database is a picture of the application domain
• Its schema is a model of the static structures of the domain
• Its data describe the current state (or suite thereof) of the domain
The role of data in business applications
2. The database is designed independently of the application programs
The database is designed before the application programs
3. The database schema evolution translates the evolution of the functional requirements
Axioms on databases
4. The database is described by (at least) two schemas:
• the conceptual schema: abstract, platform-independent
formalism: ER model, conceptual UML class diagrams
• the logical schema: concrete, platform-dependent
formalism: SQL2, Java classes
There exists a bidirectional mapping between both.
6
1. The axioms often are ignored by developers
- ignore = how interesting! I didn't know them
- ignore = I know them but they do not suit my way of working
The role of data in business applications
3. The biggest violation of the axioms concern the existence and role of the conceptual schema
Meta-axioms on axioms on databases
7
Understanding data semanticsExperimental approach and first conclusions
8Preliminary question
C400B512S144
C1
Darwen Owens Garcia
C2
London NY Madrid
C3
T
C400B512S144
CustID
Darwen Owens Garcia
Name
London NY Madrid
City
CUSTOMER
C400 Darwen LondonB512 Owens NYS144 Garcia Madrid
C
T
To what extent does each of these data setsexpresses the semantics of data?
Same data, different structures
9Motivating example. 1. Reading data from a COBOL file (1970)
application code (COBOL)
WORKING-STORAGE SECTION.01 CUSTOMER. 02 CustID PIC X(12). 02 Name PIC X(60). 02 City PIC X(40).
CustID
Name
City
CUSTOMER
external file
SELECT FILE1 ASSIGN TO "FILE1.DAT"ORGANIZATION IS INDEXEDACCESS MODE IS DYNAMICRECORD KEY IS RKEY.
FD FILE1.01 REC. 02 RKEY PIC X(12). 02 RINFO PIC X(100).
C400B512S144
RKEY
Darwen London Owens NY Garcia Madrid
RINFO
REC
REC
RKEYRINFO
CUSTOMER
CustIDNameCity
B512
CustID
Owens
Name
NY
City
CUSTOMER READ FILE1 INTO CUSTOMER.
10Motivating example: 1. Reading data from a COBOL file (1970)
REC
RKEYRINFO
CUSTOMER
CustIDNameCity
Where has data semantics been defined?
• In file description (10%) - [unique key, key data type]
• In application code (93%).
10%93%
11Motivating example. 2. Reading data from an RDB (1980+)
Relational DB
create table CUSTOMER( CustID char(12) not null, Name char(60) not null, City char(40) not null, primary key (CustID)).
CustID
Name
City
CUSTOMER
C400B512S144
CustID
Darwen Owens Garcia
Name
London NY Madrid
City
CUSTOMER
application code (C)
string v1;string v2;string v3;
v1
v2
v3
select * into v1,v2,v3 from CUSTOMER where CustID = 'B512'v1 v2 v3
B512 Owens NY
v1 CUSTOMER
CustIDNameCity
v2
v3
12Motivating example: 2. Reading data from an RDB (1980+)
Where has data semantics been defined?
• In DB schema (100%)
• In application code (3%) - [data type].
v1 CUSTOMER
CustIDNameCity
v2
v3
3% 100%
13What does data semantics mean?
A tentative practical definition
Data semantics is the knowledge defined by all the
non technical,
domain-dependent,
information
that allows us to understand, to use and to manage the data.
14Where can we find traces of data semantics?
data
DB schema
Applicationprogram
in the application code (reading from file)
in the DB schema (reading from DB)
15
1. Expressiveness: DDL is the most appropriate language to declare data structures and constraints
2. Language independence: DDL is independent of application programming languages
6. Stability. The schema must be changed only when the application domain evolve.
3. Uniqueness: the schema is unique and centralized
4. Integration with data: the schema is a part of the database (no risk to loose it!))
5. Program independence: the schema is independent of application programs
1. Expressiveness: DDL is the most appropriate language to declare data structures and constraints
A first (trivial) observation
2. Language independence: DDL is independent of application programming languages
6. Stability. The schema must be changed only when the application domain evolve.
3. Uniqueness: the schema is unique and centralized
4. Integration with data: the schema is a part of the database (no risk to loose it!))
5. Program independence: the schema is independent of application programs
It is best to express data semantics in the database schema
16
Only data structures are explicit in application programs:
• record name
• field name
• field data type
However, things are not always that simple (e.g.,COBOL files)
Additional constraints generally are controlled by the application code:
• where?
• in which way?
• in all the modules processing the data?
Understanding data semantics by analyzing the program code can be much complex than expected.
17
Only standard integrity constraints can be coded through the DDL (SQL2):
• not null
• uniqueness
• referential integrity
However, things are not always that simple (e.g., RDB)
Additional constraints must be coded through generic means:
• check predicates
• triggers
• store procedures
Understanding data semantics by reading the database schema can be less easy than
expected.
18
Data models
19Data models: abstraction hierarchy
Coding SQL-DDL code
Physical design
Logical design
Information analysis
Userrequirements
Conceptualschema
Logical (RDB)schema
Physical (DB2)schema
Reminder on the database design process - The standard view
20999. Data semantics and data models
Conceptual models
• ER (*)• UML class diagrams
Logical models
• Record oriented models: • files • legacy DBMS (IMS, CODASYL) • RDB (*)
• Key-Value models: • NoSQL (*)• CSV
• Structured object models: • OO• NoSQL• Json (*)• XML
The way data semantics is expressed in a database depends on its data model
21ER conceptual model
Abstract, platform-independent information description
The world is perceived as:- sets of entities,- properties that characterize entities- relationships holding between entities
A conceptual schema can be translated into several logical, DBMS-dependent, schemas
1-10-N place
ORDER
OrdIDDateOrdAccount
id: OrdID
CUSTOMER
CustIDNameCity
id: CustID
22
data
metadata
Relational data model (schema-based, 1NF)
Examples: Oracle, DB2, SQL Server, MySQL, PostgreSQL, etc.
• Domain-dependent schema• Schema and data are hierarchically distinct• Values are aggregated into rows• The semantics is explicit in the schema (part of!)• The semantic is managed/controlled by the DBMS
C400B512S144
CustID
Darwen Owens Garcia
Name
London NY Madrid
City
-124 5509 0
Account
23
meta-metadata
metadata
data
ENTITY
903179031790317903175973159731597315973166830668306683066830
ATTRIBUTE
CustID Name City Account CustID Name City Account CustID Name City Account
VALUE
C400 Darwen London -124 B512 Owens NY 5509 S144 Garcia Madrid 0
Key-Value data model (schema-less, triples, 1NF)
Examples: Oracle NoSQL, BerkeleyDB, Voldemort, Riak, Redis
• Domain-independent schema• Metadata mixed with data • Elementary Key-Value• The semantics is explicit in the data• The semantics is managed/controlled by application programs or middleware
24
data
metadata
{"CustID": "C400", "Name": "Darwen","City": "London", "Account": 124} {"CustID": "B512", "Name": "Owens", "City": "NY", "Account": 5509} {"CustID": "S144", "Name": "Garcia", "City": "Madrid", "Account": 0}
903175973166830
meta-metadata
ENTITY ATTRIBUTES
Structured object data models (schema-less, NF2)
Examples: CouchDB, MongoDB (BSON), SimpleDB
• Domain-independent schema• Metadata mixed with data• Aggregated Key-Value into objects (here in Json) • The semantics is explicit in the data• The semantic is managed/controlled by application programs or middleware
25
Tracing data semantics
26In the real world, where is semantics expressed?
We have identified two places: DB schema and application code.
Are there other places?
27Architectural framework
data
DB schema
Applicationprogram
O/RMapping
class schema
User interface- data structure- labels- help, error messages)
Application code- data structures- procedural code)
Class schema
DB logical schema- global schema- views
Data
Doc
Documentation (text, structured, ontology)
Object/Relational mapping
28Semantics in the documentation
data
DB schema
Applicationprogram
O/RMapping
class schema
Doc
Documentation (text, structured, ontology)
Functional documentation (should include the conceptual schema)
Technical documentation (should include the logical schema)
Drawback the documentation often is
• obsolete, • incomplete, • inconsistent• missing
298. Semantics in the DB schema
data
DB schema
Applicationprogram
O/RMapping
class schema
Doc
DB logical schema- global logical schema- views
The logical schema is DBMS-dependent.
It is a more or less faithful implementation of the conceptual schema.
Some views can be more detailed than the logical schema.
Drawbacks• not a conceptual schema• additional constraints not always trivial to
identify and to understand
3010. Semantics in the class schema
data
DB schema
Applicationprogram
O/RMapping
class schema
Class schema
Doc
DB logical schema
T
Bidirectional relation/object transformation.
Solving the impedance mismatch problem
The class schema seen as the domain model.
It is implemented into a relational database, which ensures object persistence.
The DB schema itself is hidden and may bear little semantics.
Drawbacks• inappropriate formalism• poor change propagation mechanism (if any)• semantics in the application and not in the DB• data model not easily shared by several
applications
3111. Semantics in the application code
data
DB schema
Applicationprogram
O/RMapping
class schema
Application code- data structures- procedural code
Doc
Internal data structures may be more explicit that theDB schema.
Data integrity constraints checked by the application code.
Understanding data semantics from the wayprograms process the data.
However, program analysis is far from trivial:• size (millions of LOC)• architectural complexity• algorithmic complexity• data flow complexity• creative data processing
Drawbacks• redundancies (a constraint may be checked in
many places)• distributed traces (potential inconsistencies)
3212. Semantics in the GUI
data
DB schema
Applicationprogram
O/RMapping
class schema
User interface- data structure- labels- help, error messages)Doc
The UI often is a view on a part of the database.
This view is intended for users user friendly.
Provides useful hints about the constraints and meaning of data:
• data structure (data types, aggregates)
• explicit labels
• sample data
• informative help and error messages
Drawbacks• distributed control (potential inconsistencies)• does not cover all the database objects
3313. Semantics in the data (record-oriented models)
data
DB schema
Applicationprogram
O/RMapping
class schema
Data
Doc
In standard models
Data analysis: finding relationships among data
• uniqueness
• data types
• inclusion properties (foreign keys)
• etc.
Main strategy• validating hypotheses
3413. Semantics in the data (alternative models)
data
DB schema
Applicationprogram
O/RMapping
class schema
Data
Doc
In alternative (schema-less) models
Metadata extraction
But also data analysis as in standard models
Experience• none. Too new.
35
Recovering hidden data semantics:database reverse engineering
36
Definition
DB reverse engineering
Reverse engineering a piece of software consists, among others, in recovering or reconstructing its functional and technical specifications, starting mainly from the source text of the programs. Recovering these specifications is generally intended to redocument, convert, refactor, maintain or extend existing applications.
Database reverse engineering is that part of Information System Engineering that addresses the problems and techniques related to the recovery of the conceptual and logical schemas of files and databases of existing systems.
37
DB reverse engineering methodology
DB reverse engineering
Full project
Pilote
Conceptualization
Logical extraction
Physical extraction
Sourcemanagement
Projectplanning
Conceptualschema
Logical (RDB)schema
38
DB reverse engineering methodology
DB reverse engineering
Full project
Pilote
Conceptualization
Logical extraction
Physical extraction
Sourcemanagement
Projectplanning
Others
UI analysis
Class analysis
Prog. analysis
Data analysis
Sch. analysis
Normalization
Untranslation
De-optimization
39
Is data semantics recovery that important, actually?
40
Yes
Definitely!
41Can you prove it? At least I can show you an example
Example: database application migration
Porting a complete existing application, or some of its components, on another, generally
more modern, platform.
For a database: changing its DMS. A popular example: migrating the legacy set of files of
a business application to a RDBMS.
Two main approaches :
• physical approach
• semantic approach
42Physical database migration
Database migration
The physical, or one-to-one migration strategy is the cheapest but also the worst
approach since it deeply degrades the final structure.
Requires no knowledge on data semantics Very popular
Physicalextraction
Physical (file)schema
COBOL code SQL-DDL code
Coding
Physical (DB2)schemaTransform
43Physical database migration
physical (one-to-one) migration
SELECT CLIENT ASSIGN TO "CUST.DAT"ORGANIZATION IS INDEXEDRECORD KEY IS CUST_ID.FD CUST-FILE.01 CUSTOMER. 02 CUST-ID PIC X(12). 02 CUST-INFO PIC X(80). 02 CUST-HIST PIC X(1000).
Create table CUSTOMER( CUST_ID char(12) not null, CUST_INFO char(80) not null, CUST_HIST char(1000) not null, primary key (CUST_ID));
=
=
CUSTOMER
CUST-ID: char (12)CUST-INFO: char (80)CUST-HIST: char (1000)
id: CUST-ID
CUSTOMER
CUST_ID: char (12)CUST_INFO: char (80)CUST_HIST: char (1000)
id: CUST_ID
no added value
44Semantic database migration
Database migration
Semantic approach: based on an in-depth understanding of the semantics of source data.
Provides a high quality result. Strong basis for the future.
Requires a complete, up to date, knowledge of the DB
Physicalextraction
Physical (IDMS)schema
Logical (DBTG)schema
Conceptualschema
Logical extraction
Conceptual-ization
IDMS-DDL code SQL-DDL code
Coding
Physicaldesign
Logicaldesign
Logical (RDB)schema
Physical (DB2)schema
Conceptualschema
Reverse Engineering
COBOL code SQL-DDL code
Coding
Physicaldesign
Logicaldesign
Logical (RDB)schema
Physical (DB2)schema
45Semantic database migration (1)
semantic migration (refinement)
SELECT CLIENT ASSIGN TO "CUST.DAT"ORGANIZATION IS INDEXEDRECORD KEY IS CUST_ID.FD CUST-FILE.01 CUSTOMER. 02 CUST-ID PIC X(12). 02 CUST-INFO PIC X(80). 02 CUST-HIST PIC X(1000).
+
CUSTOMERCUST-ID: char (12)CUST-INFO: compound (70)
NAME: char (20)ADDRESS: char (40)STATUS: char (10)
CUST-HIST-PURCH[0-100] array: compound (10)ITEM: num (5)TOTAL: num (5)
id: CUST-IDid(CUST-HIST-PURCH):
ITEM
1-10-100 record
CUSTOMER
CUST-ID: char (12)CUST-INFO: compound (70)
NAME: char (20)ADDRESS: char (40)STATUS: char (10)
id: CUST-ID
CUST-HIST-PURCH
Index: index (4)ITEM: num (5)TOTAL: num (5)id: record.CUSTOMER
ITEMid': record.CUSTOMER
Index
46Semantic database migration (2)
semantic migration (SQL translation)
1-10-100 record
CUSTOMER
CUST-ID: char (12)CUST-INFO: compound (70)
NAME: char (20)ADDRESS: char (40)STATUS: char (10)
id: CUST-ID
CUST-HIST-PURCH
ITEM: num (5)Index: index (4)TOTAL: num (5)id: record.CUSTOMER
ITEMid': record.CUSTOMER
Index
No more than 100 CUST_HIST_PURCHper CUSTOMER
CUSTOMER
CUST_IDCUS_NAMECUS_ADDRESSCUS_STATUS
id: CUST_ID
CUST_HIST_PURCH
CUST_IDITEMCINDEXTOTALid: CUST_ID
ITEMid': CUST_ID
CINDEXref: CUST_ID
Create table CUSTOMER( CUST_ID char(12) not null, CUST_NAME char(28) not null, CUST_ADDRESS char(60) not null, CUST_STATUS char(2) not null, primary key (CUST_ID));
Create table CUST_HIST_PURCH( CUST_ID char(12) not null, ITEM char(10) not null, CINDEX smallint not null check(CINDEX <= 100), TOTAL smallint not null, primary key (CUST_ID,ITEM), unique (CUST_ID,CINDEX), foreign key (CUST_ID) reference CUSTOMER);
Normalized DB
47Database migration - Synthesis
Create table CUSTOMER( CUST_ID char(12) not null, CUST_NAME char(28) not null, CUST_ADDRESS char(60) not null, CUST_STATUS char(2) not null, primary key (CUST_ID));
Create table CUST_HIST_PURCH( CUST_ID char(12) not null, ITEM char(10) not null, CINDEX smallint not null check(CINDEX <= 100), TOTAL smallint not null, primary key (CUST_ID,ITEM), unique (CUST_ID,CINDEX), foreign key (CUST_ID) reference CUSTOMER);
Create table CUSTOMER( CUST_ID char(12) not null, CUST_INFO char(80) not null, CUST_HIST char(1000) not null, primary key (CUST_ID));
physical migration
semantic migration
48Evolution
new application: compute total sales per item
CUSTOMER
CUST-ID: char (12)CUST-INFO: char (80)CUST-HIST: char (1000)
id: CUST-ID
?
• where is the required information?
• how to extract it from the CUSTOMER table?
• who will develop the (C, Java, VB) program?
• … and when?
Select ITEM, sum(TOTAL)from CUST_HIST_PURCHgroup by ITEM;
• clearly visible + documentation if needed
• just name the columns
• by any non expert
• immediately, 2 minutes
CUST_HIST_PURCH
CUST_IDITEMCINDEXTOTALid: CUST_ID
ITEMid': CUST_ID
CINDEXref: CUST_ID
CUSTOMER
CUST_IDCUS_NAMECUS_ADDRESSCUS_STATUS
id: CUST_ID
49
Summary and conclusions
50
• Theories (e.g., text books) teach that the conceptual schema must be the unique expression of data semantics. In an ideal world, the conceptual schema exists, and all the other artefacts (DB schemas, UML diagrams, views, class schema, programs, UI) derive from it and capture each a part of this semantics.
Some mundane observations
• Identifying, extracting, understanding and merging these traces to rebuilt the conceptual schema are the very goals of database reverse engineering.
• However, the real world doesn't learn from theories. Most often, the conceptual schema does not exist so that only the other artefacts bear traces of the data semantics.
51Cultural aspects of data semantics expression
1. Small personal application
Mainly non-professional developers. Intuitive, bottom-up, incremental development. Weak culture in DB.
Data semantics: in the UI, in application code
2. Database (record-oriented) data-intensive processing
Professional developers. Disciplined, top-down development. Strong culture in DB.
Data semantics: in the DB schema (including additional constraints).
3. OO data-intensive processing
Professional developers. OO minded. Disciplined, top-down development. Weak culture in DB.
Data semantics: in the class schema (through O/RM middleware).
4. Big data
(Semi-)Professional developers. Low complexity applications.RDB discarded as old-style (however NewSQL DBMS are lurking!)
Data semantics: simple, loose (few constraints); metadata in data
52
1950 - 1975: file-oriented processing
Semantics in record schema and application code
Evolution of data semantics expression
1968 - 1990: hierarchical/network database processing
Semantics in DB schema
1980 - ?: relational database processing
Semantics in DB schema
1990 - 2000: object-oriented DB processing
Semantics in DB schema and application code (methods)
2000 - ?: object-relational DB processing
Semantics in DB schema
2000 - ?: O/RM processing
Semantics in class schema
2011 - ?: NewSQL DB processing
Semantics in DB schema
2005 - ?: NoSQL DB processing
Semantics in data and in application code
prog
DB
DB
prog
DB
prog
prog
DB
Quality of DS representation
53
Quite often, developers see the database as a mere repository for the data used and created by programs:
• "the database offers persistence services for the business logic layer"
• "the database is an implementation of the program classes"
Some conclusions
This view entails much problems when long term maintenance and evolution are concerned. When the program changes, the database schema often must be modified accordingly, even if its semantics does not change.
The view of the database as a model of the application domain ensures a great stability of business systems.
So, the database is directly dependent on the current state of program architecture.
It makes the joy of researchers in system evolution but lets the practitioners less enthousiast.
Is the database culture still living among today developers?
54
Thanks
55
56
57
Abstract of the lecture
The role of databases may sometimes appear controversial since they are mere basic services for a significant part of the the software engineering community (the transparent "persistence layer") while they are the central component of business application for the database community. In this lecture, we examine the evolution of the balance database/program both in time (from the early sixties to a foreseenable future) and in space (technologies, communities) from the data semantics point of view. In particular we analyze and compare how and where data semantics has been located and implemented in each of these contexts. Current development practices tend to migrate semantics from the database (as was usual in the eighties and nineties) to the application logic (e.g., O/RM, NoSQL DB managers), a trend that may be seen of regression that reminds us the infancy of business application development where files were dedicated to one application. Finally, the lecture defines how data semantics can be recovered in these scenarios.