maime: a maintenance manager for etl processesceur-ws.org/vol-1810/dolap_paper_08.pdfmaime: a...

MAIME: A Maintenance Manager for ETL Processes

Dariuš Butkevicius∗

Vilnius [email protected]

Philipp D. FreibergerAalborg University

{pfreib15

Frederik M. HalbergAalborg University

fhalbe12Jacob B. HansenAalborg University

jh12

Søren JensenAalborg University

sajens12

Michael TarpAalborg University

mtarp09}@student.aau.dk

ABSTRACTThe proliferation of business intelligence applications movesmost organizations into an era where data becomes an es-sential part of the success factors. More and more busi-ness focus has thus been added to the integration and pro-cessing of data in the enterprise environment. Develop-ing and maintaining Extraction-Transform-Load (ETL) pro-cesses becomes critical in most data-driven organizations.External Data Sources (EDSs) often change their schemawhich potentially leaves the ETL processes that extract datafrom those EDSs invalid. Repairing these ETL processesis time-consuming and tedious. As a remedy, we proposeMAIME as a tool to (semi-)automatically maintain ETLprocesses. MAIME works with SQL Server Integration Ser-vices (SSIS) and uses a graph model as a layer of abstractionon top of SSIS Data Flow tasks (ETL processes). We intro-duce a graph alteration algorithm which propagates detectedEDS schema changes through the graph. Modifications doneto a graph are directly applied to the underlying ETL pro-cess. It can be configured how MAIME handles EDS schemachanges for different SSIS transformations. For the con-sidered set of transformations, MAIME can maintain SSISData Flow tasks (semi-)automatically. Compared to doingthis manually, the amount of user inputs is decreased by afactor of 9.5 and the spent time is reduced by a factor of 9.8in an evaluation.

Categories and Subject DescriptorsInformation systems [Information integration]: Extrac-tion, transformation and loading

1. INTRODUCTIONBusiness Intelligence (BI) is an essential set of techniques

and tools for a business to provide analytics and reporting inorder to give decision support. The BI techniques and tools

∗Work done at Aalborg University

2017, Copyright is with the authors. Published in the Workshop Proceed-ings of the EDBT/ICDT 2017 Joint Conference (March 21, 2017, Venice,Italy) on CEUR-WS.org (ISSN 1613-0073). Distribution of this paper ispermitted under the terms of the Creative Commons license CC-by-nc-nd4.0

use a Data Warehouse (DW) containing the data for thedecision making of a business. A DW contains transformeddata from one or more External Data Sources (EDSs) [4]. Inorder to populate a DW, an Extract-Transform-Load (ETL)process is used. An ETL process extracts data from one ormore EDSs, transforms the data into the desired format bycleansing it (i.e., correcting or removing corrupt/inaccuratedata), conforming from multiple sources, deriving new val-ues, etc., and finally loads it into the target DW.

Construction of an ETL process is very time-consuming [4].Maintaining ETL processes after deployment is, however,also very time-consuming. The challenges in the mainte-nance of ETL process are very well demonstrated by thefollowing examples which we have obtained from Danishorganizations. The data warehouse solution at a pensionand insurance firm needs to cope with unknown changes inthe main pension system when weekly hotfixes are deployed.The database administrator at a facility management com-pany has to manage more than 10,000 ETL jobs that executedaily. The ETL team at an online gaming-engine vendor hasto manage the daily format changes in the web services deliv-ering sales and marketing data. The data warehouse teamat a financial institution strives between the challenges ofvariances of source data formats and a fixed monthly releasewindow for ETL programs. To summarize, when the numberof ETL programs becomes overwhelming for the IT team,management of these programs becomes time-consuming.In such a situation, unexpected changes in source systemdatabases or source system deliveries add more complexityand risk in the maintenance of the ETL programs.

The impact of an EDS schema change depends on boththe type of the EDS schema change and how ETL processesuse the changed EDSs. Maintenance of ETL processes thusrequires manual work, is very time-consuming, and is quiteerror-prone. To remedy these problems, we propose the toolMAIME which can (1) detect schema changes in EDSs and(2) semi-automatically repair the affected ETL processes.MAIME works with SQL Server Integration Services (SSIS)[5] and supports Microsoft SQL Server. SSIS is chosen as theETL platform since it is in the top three of the most usedtools by businesses for data integration [12], has an easy touse graphical tool (SSIS Designer [2]), and includes an API,which allows access to modify ETL processes through thirdparty programs like MAIME. To maintain ETL processes,we formalize and implement a graph model as a layer ofabstraction on top of SSIS Data Flow tasks. By doing so,we only have to modify the graph which then automaticallymodifies the corresponding SSIS Data Flow task. Running

ETL processes is often an extensive and time-consumingtask, and a single ETL process not being able to executecould cause a considerable amount of time wasted since theadministrator would have to repair the ETL process and runthe ETL processes once again. Based on this observation,MAIME can be configured to repair ETL processes even ifthis requires deletion of parts/transformations of the ETLprocesses.

In an evaluation MAIME is shown to be able to suc-cessfully repair ETL processes in response to EDS schemachanges. For the implemented set of transformations, a com-parison between resolving EDS schema changes in MAIMEand doing it manually in the SSIS Designer tool shows thatMAIME is on average 9.8 times faster and required 9.5 timesless input from the user to maintain.

The rest of the paper is organized as follows. Section 2provides an overview of MAIME and how it detects EDSschema changes. Section 3 formalizes the graph model anddescribes its usage. Section 4 describes how graph alter-ations are done. Section 5 describes the implementation.Section 6 shows how MAIME compares to doing the main-tenance manually. Section 7 covers related work. Section 8concludes and provides directions for future work.

2. OVERVIEW OF MAIMEIn this section, we give an overview of the SSIS com-

ponents covered by our solution. Then, we explain howMAIME detects EDS schema changes. Finally, we give adescription of the architecture of MAIME.

Knight et al. [5] provide an overview of the SSIS architec-ture. To briefly summarize, a package is a core component ofSSIS which connects all of its tasks together. The package isstored in a file which the SSIS engine can load to execute thecontained control flows. A control flow of a package controlsthe order and execution of its contained tasks. This workfocuses on Data Flow tasks. A Data Flow task can extractdata from one or more sources, transform the data, and loadit into one or more destinations. It can be observed that theactions performed in a Data Flow task thus resemble thoseof the general notion of an ETL process. We therefore re-gard a Data Flow task as an ETL process. A Data Flow taskcan make use of several types of transformations referred toas Data Flow components in SSIS. The current prototype ofMAIME covers the following subset of common transforma-tions: Aggregate, Conditional Split, Data Conversion, De-rived Column, Lookup, Sort, and Union All. To extract andload data, we use OLE DB Source and OLE DB Destina-tion. For convenience, we extend the term“transformations”to also cover OLE DB Source and OLE DB Destination eventhough they do not transform data.

We now consider how to maintain ETL processes usingthese transformations. The first problem is to detect EDSschema changes. We do not want MAIME to rely on thirdparty tools for this. One possibility is then to use triggers.They would, however, need to be created first and for manysource systems, there would be many of them to handle bothdeletions, renames, and additions for hundreds of tables.Further, they would fire for each individual modification andnot only after all modifications are done. Instead, we there-fore just extract metadata from SQL Server’s InformationSchema Views with plain SQL statements (other DBMSstypically offer something similar such that support for themalso could be added). We thus store a snapshot of how the

EDS Change Manager

EDS Change Manager

Log

Metadata Store

Configuration

Maintenance Manager

Maintenance Manager

GUIGUI

EDS

1

1...n

1

1...n

Figure 1: Architecture of MAIME.

source databases look and when MAIME is run, it creates anew snapshot and compares that to the previous snapshotto detect which changes have occurred.

Figure 1 shows the architecture of MAIME. We first pro-vide short descriptions for MAIME’s surrounding elementsthat are used by the core MAIME components. EDSs corre-spond to the external data sources of which schema changesare detected. In Figure 1, the Metadata Store correspondsto a local directory. Each captured EDS metadata snapshotis stored as a JSON file in the Metadata Store. The Logis a collection files that log every change done by MAIMEso that an administrator can analyze the maintenance. TheConfiguration is a JSON file which stores the user-definedconfigurations.

The EDS Change Manager (ECM) captures metadatafrom the EDSs by using the Information Schema Views.The metadata snapshots are stored in the Metadata Store.Before repairing an ETL process, the current snapshot iscompared with the previous snapshot. As a result, a listof EDS schema changes is produced which is accessible tothe Maintenance Manager (MM). The core logic of MAIMEresides in the MM. The MM loads specifications of ETL pro-cesses (i.e., SSIS Data Flow tasks) and creates correspondinggraphs. One graph is responsible for updating one underly-ing ETL process. The MM contains multiple graphs andoperates on them when the graph alteration algorithm (ex-plained in Section 4.2) is called for the found EDS schemachanges. Modifications made by the graph alteration algo-rithm to the graphs are done semi-automatically since theuser can be prompted. How exactly this is done dependson the configurations described in Section 4.1. The GUI isaccessed by an administrator to maintain ETL processes. Itinvolves: (1) Selecting ETL processes to maintain, (2) ad-justing administrator configurations, (3) confirming or deny-ing changes by answering to prompts, (4) visualizing appliedchanges to the ETL processes, and (5) displaying log entriesfor the applied changes.

3. GRAPH MODELTo analyze ETL processes and the effects of EDS schema

changes, the ETL processes are modeled using graphs whichprovide a level of abstraction over the SSIS packages. Whenwe make a change in our graph, corresponding changes arealso applied to the underlying SSIS package. Afterwards,we save the SSIS package which is now in an executablestate. An advantage of using graphs for modeling an ETL

process lies in the capability of easily representing dependen-cies between columns for all transformations and handlingcascading changes using graph traversal. In this section, wedescribe our graph model in details.

Each ETL process is represented as an acyclic propertygraph G = (V,E) where a vertex v ∈ V represents a trans-formation and an edge e ∈ E represents that columns aretransferred from one transformation to another. A propertygraph is a labeled, attributed, and directed multi-graph [10].Each vertex and edge can have multiple properties. A prop-erty is a key-value pair. To refer to the property name ofvertex v1, we use the notation v1.name. The set C rep-resents all columns used in an ETL process (i.e., columnsfrom an EDS and columns created during the ETL process).Each column c ∈ C is a 3-tuple (id, name, type) where id isa unique number, name is the name of the column, and typeholds the data type of the column. id was included becausename and type are not enough to uniquely identify a col-umn. Each edge e ∈ E is a 3-tuple (v1, v2, columns) wherev1, v2 ∈ V and columns ⊆ C is the set of the columns be-ing transferred from v1 to v2. Putting columns on the edgeis particularly advantageous for transformations which canhave multiple outgoing edges where each edge can transfera different set of columns, such as Aggregate in SSIS.

A vertex v representing a transformation has a set ofproperties depending on the type of transformation. Theproperties all vertices have in common are: name, type,and dependencies. The only exception is a vertex for anOLE DB Destination which does not have the dependenciesproperty. name is a unique name used to identify vertexv. type denotes which kind of transformation v represents(e.g., Conditional Split or Aggregate). dependencies showshow columns depend on each other. If, for example, weextract the column c from an EDS and use an Aggregatetransformation that takes the average of c and outputs c′,we say that c′ 7→ {c}. In the case of the Aggregate trans-formation, c′ originates from c and is therefore dependenton c. Any modifications such as deletion of c or a modi-fication of its data type can result in a similar change toc′. Formally, dependencies is a mapping from an outputcolumn o ∈ C to a set of input columns {c1, . . . , cn} ⊆ C.We say that o is dependent on {c1, . . . , cn} and denote thisas: o 7→ {c1, . . . , cn}. The output columns are defined asthe columns that a vertex sends to another vertex throughan outgoing edge, such as an OLE DB Source transferringcolumns to an Aggregate. The input columns are definedas the columns that a vertex receives from another vertexthrough an incoming edge. The main purpose of dependen-cies is to detect whether an EDS schema change has any cas-cading effects. Due to this, the graph contains some trivialdependencies if a transformation does not affect a column,e.g., a dependency such as: c 7→ {c} where c is dependenton itself. One example of an output column depending onmultiple input columns is a Derived Column transformationwhere a derived output column o is dependent on the inputcolumns i1 and i2 if they were used to derive the value ofo. Thus o 7→ {i1, i2} showing that if an EDS schema changeaffects i1 or i2 then it may also affect o.

Additional properties of a vertex are defined by the typeof the vertex. For instance, v.aggregations is a necessaryproperty for Aggregate, whereas OLE DB Source does notuse aggregations and does not have this property. Table 1shows all properties specific to each transformation, and the

Table 1: Transformations, their properties, andnumber of allowed incoming and outgoing edges.

Transformation Specific properties In OutOLE DB Source database, table,

and columns0 1

OLE DB Destin. database, table,and columns

1 0

Aggregate aggregations 1 manyConditional split conditions 1 manyData conversion conversions 1 1Derived column derivations 1 1Lookup database, table,

joins, columns, andoutputcolumns

1 2

Sort sortings andpassthrough

1 1

Union all inputedges andunions

many 1

number of incoming and outgoing edges. The following de-scribes the definitions of properties in more detail and howdependencies is specified for each type of transformation.

An OLE DB Source is used to extract data from a tableor view of an EDS. It provides data for upcoming transfor-mations and is thus represented by a vertex with no in-coming edges. An OLE DB Source has some additionalproperties. database is the name of the database data isextracted from. table is the name of the table data is ex-tracted from. columns is the list of columns extracted fromthe table. dependencies for OLE DB Source is trivial as foreach column c ∈ columns, c 7→ ∅. Each column is dependenton nothing in the OLE DB Source, as this is the first timethe columns appear in the graph.

An OLE DB Destination represents an ending point inthe graph and is responsible for loading data into a DW.OLE DB Destination has the same properties as OLE DBSource, here representing where columns are loaded to. SinceOLE DB Destination does not have any outgoing edges,dependencies does not exist for OLE DB Destination.

An Aggregate applies aggregate functions such as SUMto values of columns which results in new outputs withthe aggregated values. Formally, the only property spe-cific to an Aggregate vertex is defined as aggregations ={(f1, input1, output1, dest1),..., (fn, inputn, outputn, destn)}where fi ∈ {COUNT,COUNT DISTINCT,GROUP BY,SUM,AV G,MIN,MAX} is the aggregate function, inputi∈ C is the input column that the ith output column is com-puted from, and outputi ∈ C is the result of the ith aggrega-tion and output of aggregate i, respectively. desti ∈ V is thevertex receiving the output column outputi. A given desti-nation dest can appear in multiple tuples of aggregations,which shows that the dest receives multiple columns througha single edge. For each tuple i in aggregations, we have thedependency outputi 7→ {inputi}.

A Conditional Split applies expressions to rows andcan thereby route them to different destinations based onwhich conditions they satisfy. Expressions are for exampleSUBSTRING(title, 1, 3) == ”Mr.” or salary > 30000.Formally, the only property specific to the Conditional Splitvertex is defined as conditions = {(expr1, p1, dest1), . . . ,(exprn, pn, destn)} where expri is a predicate that speci-fies which rows are routed to desti, pi ∈ N+ is a prioritywhich indicates in which order the conditions are evaluated

in (where 1 is the highest priority). The set of priorities{p1, . . . , pn} is a gap-free series of numbers that range from1 to n, where n is the number of conditions. desti ∈ V isthe vertex that receives rows (that have not been taken bya higher priority condition) for which expri holds. In Con-ditional Split the set of output columns on each outgoingedge is equal to the set of input columns. This is because aConditional Split transformation is not able to add, delete,or otherwise change columns, but is only able to filter rows.Thus, each column depends on itself.

A Data Conversion casts the value of a column into anew column with the new data type. Formally, the onlyspecific property of the data conversion vertex is defined asconversions = {(input1, output1), . . . , (inputn, outputn)}where inputi ∈ C is the input column of the vertex andoutputi ∈ C is a new column that has been created throughthe Data Conversion from inputi. The specific conversionsof data types are present in the type of outputi. A DataConversion creates a new column rather than replacing theexisting column and thus both inputi and outputi are out-put columns. dependencies is simple for Data Conversion,as a dependency is made for each new outputi such thatoutputi 7→ {inputi} for all tuples in conversions and sinceall input columns are sent to next vertex, each depends onitself.

A Derived Column creates new columns based on inputcolumns and expressions applied to those inputs. Formally,the only property specific to the Derived Column vertexis defined as derivations = {(expr1, output1), . . . , (exprn,outputn)} where expri is an expression used for computingthe new values of outputi, and outputi ∈ C is a newly cre-ated column. dependencies for Derived Column is similar toData Conversion in that each input column is dependent onitself. For all output columns in derivations we define thedependency outputi 7→ {ci,1, . . . , ci,j} where {ci,1, . . . , ci,j}is the set of all columns used in expri.

A Lookup extracts additional data from a database byequi-joining with given input columns. Formally, the prop-erties of a Lookup vertex are defined as follows: databaseis the name of the database to lookup additional columnsfrom. table is the name of the table used. columns ⊆C represents all columns in the table. outputcolumns ⊆columns is the columns extracted from the table. joins ={(input1, lookup1), . . . , (inputn, lookupn)} where joins rep-resents the equi-join of the Lookup. inputi is a column fromthe preceding vertex used for the join condition and lookupiis a column from columns. The equi-join is used to extractthe columns in outputcolumns. Each input column has adependency to itself. Each new column in outputcolumns isdependent on all the input columns used in the join con-ditions of the Lookup. In other words, for each outputcolumn output ∈ outputcolumns, we have that output 7→{input1, . . . , inputn} where inputi is the input column ofthe ith tuple in joins.

A Sort sorts the input rows in either ascending or de-scending order and creates new output columns for the sortedrows. It is possible to sort on multiple columns where a cer-tain priority has to be given. Formally, the properties ofa Sort vertex are defined as follows. sortings = {(input1,output1, sorttype1, order1), . . . , (inputn, outputn, sorttypen,ordern)} where inputi, outputi ∈ C, sorttypei ∈ {ascending,descending}, and orderi ∈ N+ indicates in which order thecolumns are sorted (where 1 is the highest priority). The set

Store

ID: int (PK)Address: nvarchar(60)

Sale

ID: int (PK)CustomerID: int (FK)StoreID: int (FK)TotalAmount: money

Person

ID: int (PK)Name: nvarchar(30)Age: int

Figure 2: Example of an EDS schema.

of orders {order1, . . . , ordern} is a gap-free series of num-bers from 1 to n. passthrough = {c1, . . . , cj} is the set ofcolumns passed through the transformation. This does notinclude columns that are used for sorting. For dependencies,every output column is dependent on itself.

A Union All combines rows from multiple vertices. For-mally, the properties of a Union All vertex are defined asfollows. inputedges = (e1, . . . , ej) where ei is the ith incom-ing edge of the Union All and j is the amount of incomingedges. unions = {(output1, input1), . . . , (outputn, inputn)}where inputi = (c1, . . . , cj). Each element in unions showshow multiple input columns are combined into a single out-put column. ci ∈ C ∪ ε where ε is a convention used toindicate that for a given union, no input is taken from thecorresponding edge. For example, the tuple (outputi, inputi)where inputi = (c1, ε, c3) indicates that the unioned outputiuses c1 from e1, c3 from e3, but no column is used from e2.outputi is the column generated by unioning the columnsin the ith input tuple. A column from a table can onlybe part of a single union. For the ith tuple of unions,outputi 7→ {c1, . . . , cn} such that the new column, whichis a result of the union, is dependent on all columns thatwere used for the union. This dependency is derived for alltuples in unions.

Example 1. Figure 2 shows the schema of an EDS calledSourceDB. Now consider the SSIS Data Flow task shown inFigure 3 where– OLE DB Source extracts the columns ID, Name, andAge from the Person table.– Lookup extracts the TotalAmount from the Sale table byjoining Sale.CustomerID with Person.ID.– Derived Column derives the new column AmountTimes10which is derived from the derivation (TotalAmount * 10,AmountTimes10).– Conditional Split splits the rows into two directions basedon {(Age > 40, 1, Aggregate),(TotalAmount > 10000, 2,OLE DB Destination)}.– Aggregate computes into AvgAmount the average of Amount-Times10 when grouping by Age.– OLE DB Destination loads ID, Age, and AmountTimes10into the DW table PersonSalesData.

Figure 3: An ETL process.

Columns = {(1, ID, int),(2, Name, string),(3, Age, int),(4, TotalAmount, currency),(5, AmountTimes10, currency)}

Name OLE DB Source Database = SourceDB Table = Person Columns = { ID, Name, Age}Dependencies = {ID ØName ØAge Ø }

Columns = {(1, ID, int),(2, Name, string),(3, Age, int)}

1 2 3

Name Lookup Database = SourceDB Table = Sale Columns = { ID, CustomerID, StoreID, TotalAmount}OutputColumns = {TotalAmount}Joins = {(ID, CustomerID)}Dependencies = {ID {ID}Name {Name}Age {Age}TotalAmount {ID}}

Name = Derived Column Derivations = { TotalAmount * 10", AmountTimes10)}Dependencies = {ID {ID}Name {Name}Age {Age}TotalAmount {TotalAmount}AmountTimes10 {TotalAmount}}

Columns = {(1, ID, int),(2, Name, string),(3, Age, int),(4, TotalAmount, currency)}

4

Name = Conditional Split Conditions = { Age > 40", 1, Aggregate), TotalAmount > 10000", 2, OLE DB Destination)}Dependencies = {ID {ID}Name {Name}Age {Age}TotalAmount {TotalAmount}AmountTimes10 {AmountTimes10}}


5

Name = OLE DB Destination Database = TargetDW Table = PersonSalesData Columns = { ID, Age, AmountTimes10 }

67

Name = Aggregate Aggregations = { (AVERAGE, AmountTimes10, AvgAmount, OLE DB Destination 1),(GROUP BY, Age, Age, OLE DB Destination 1)}Dependencies = {Age {Age}AvgAmount {AmountTimes10}}

Columns = {(1, Age, int),(2, AvgAmount, currency)}

Name = OLE DB Destination 1 Database = TargetDW Table = SalesData Columns = {Age, AvgAmount }


Figure 4: A MAIME graph for the Data Flow task in Figure 3.

– OLE DB Destination 1 loads AvgAmount and Age intothe DW table SalesData.

Figure 4 shows the corresponding MAIME graph wherethere are seven vertices, i.e., one for each transformation(the types are not shown since they are obvious from thenames of the vertices). As an example of a dependency, wehave AmountTimes10 7→ {TotalAmount} in Derived Col-umn. This comes from the derivation TotalAmount * 10.This shows that any change to TotalAmount can affect Am-ountTimes10, and if TotalAmount is deleted, AmountTimes-10 can no longer be derived and therefore also needs to bedeleted.

4. GRAPH ALTERATIONIn this section, we describe how the MAIME graph is up-

dated when changes happen at the EDSs. First, we describeMAIME’s user-configurable settings. Then we describe thealgorithm that updates the graph (which in turn updatesthe underlying SSIS Data Flow task).

4.1 Administrator ConfigurationsThis section describes how an administrator can configure

which modifications MAIME is allowed to perform in case ofEDS schema changes. As an example, an administrator candefine that only renames in the EDS should be propagatedto the SSIS Data Flow tasks automatically, and MAIME

should block (i.e., not propagate) any other changes in theEDS which then have to be handled manually. If MAIME isgiven free reign, it can, however, make any number of mod-ifications to ensure that ETL processes execute successfully.In the following, we describe the configurations that an ad-ministrator can specify before executing MAIME. There aretwo types of configurations: (1) EDS schema change con-figurations, and (2) Advanced configurations. To properlyunderstand the configurations, we first explain the differentkinds of EDS schema changes. The schema changes thatwe consider are the following. Addition which representsa new column in a schema; Deletion which represents adeleted column in a schema; Rename which represents arenamed column in a schema; Data Type Change whichrepresents a column where the data type has changed (incl.properties such as length).

For an EDS schema change ch ∈ {Addition, Deletion, Re-name, Data Type Change} and vertex type t, a policyp(t, ch) ∈ {Propagate, Block, Prompt} can be specified bythe administrator. The Propagate policy defines that whenp(t, ch) = Propagate, reparation of vertices of type t is al-lowed for EDS schema changes of type ch. The Block pol-icy says that when p(t, ch) = Block then for every vertex vwhere v.type = t, the alteration algorithm (explained in Sec-tion 4.2) is not allowed to make modifications to v or to anysuccessor vsucc of v, even if p(vsucc.type, ch) = Propagate.The Prompt policy defines that the choice of whether to

block or propagate the change is deferred to runtime wherethe user is prompted to make the choice. Figure 5 showshow the administrator can set a policy for a type of an EDSschema change, or be more specific and choose a policy foreach type of vertex for a given EDS change. For example, shecan configure that for all Conditional Split vertices, deletionof columns should be propagated, but all other EDS changesshould be blocked.

AdditionProp.

AggregateProp.

RenameProp.

Data typeBlock

Cond. SplitBlock

SortProp.

DeletionBlock

AggregateBlock

Cond. SplitProp.

SortBlock

EDS Schema Change Configurations

Advanced Configurations

Allow deletion of transformations

Allow modification of expressions

Use global blocking semantics

Figure 5: Configuration of MAIME.

As can be seen in Figure 5, advanced configurations canalso be specified. MAIME is based on the idea that ETLprocesses should be repaired such that all of them can exe-cute afterwards. We now explain some more difficult casesof maintaining ETL processes and which configurations theadministrator can use for such cases. Considering the caseof a deletion of a column, we need to take into account thatboth columns and vertices can be dependent on the deletedcolumn. Columns such as a derived column is dependent onthe columns that it was derived from. Vertices are depen-dent on columns if the columns are a part of that vertex’sinput, expression, condition, or used in some other way bythe vertex.

The Use global blocking semantics option dictateswhether the alteration algorithm should terminate if the pol-icy for any vertex is Block. If Use global blocking semanticsis enabled and p(t, ch) = Block then an ETL process contain-ing a vertex of type t will not be considered for reparationwhenever the EDS schema change is of type ch.

The Allow deletion of transformations option allowsdeletion of vertices in a graph (and thus transformations ina SSIS Data Flow task). With this option enabled, MAIMEcan modify graphs to such an extent that large portions of agraph are removed, but the corresponding updated ETL pro-cess can execute. Consider, for example, if a Sort vertex usescolumn a for sorting. If a is deleted from the EDS, it wouldrender the vertex invalid. With Allow deletion of transfor-mations option enabled, the alteration algorithm attemptsto repair the process by deleting the Sort vertex and anysuccessive vertices, thereby making the process executable.The algorithm described in Section 4.2 further explains thisprinciple.

The Allow modification of expressions option allowsthe alteration algorithm to modify expressions in vertices(such as Conditional Split and Derived Column) in the eventof deletions or data type changes of columns. Consider, forexample, a Conditional Split with input columns a and band one condition with the expression a < 20 && b < 40

and assume that an EDS schema change of b being deletedoccurs. If Allow modification of expressions is enabled, thealteration algorithm attempts to update the expression intoa < 20. This would make the process run successfully with-out removing the whole expression, but the semantics isdifferent. If Allow modification of expressions is disabled,the alteration algorithm would instead remove the expres-sion. Modification of expressions is always allowed if theEDS schema change is the renaming of a column. This isbecause it is very simple to preserve the original semanticsby replacing the old name of the renamed column with thenew name in the expression.

4.2 The Graph Alteration AlgorithmThis section details how our graph model adapts to EDS

schema changes. As described above, the administrator canchoose between the Propagate, Block, and Prompt poli-cies. Only a propagation (which can also occur througha prompt) alters the graph. This section goes into depth onhow to propagate an EDS schema change through the graph.How propagation is handled for a given type of EDS schemachange is specific to each type of transformation. For spacereasons, details about the different actions are not providedhere, but are available elsewhere [3]. However, just apply-ing actions to each vertex independently is not enough, as achange in one vertex can also affect successive vertices. TheAlter-graph algorithm takes in a single EDS schema changech and our property graph G. To be able to handle multipleEDS schema changes, the Alter-graph algorithm is calledfor each EDS schema change. For simplicity, we only presentthe case where Allow deletion of transformations is enabledand neither Prompt nor data type change is used.

The order in which we traverse the graph matters as vis-iting a vertex also affects successive vertices. For this, topo-logical sorting is used on Line 1 such that when a vertex isvisited, it is guaranteed that all its predecessors have beenvisited beforehand. The list of topologically sorted verticesis referred to as L. On Lines 2–6, the algorithm handles thecase where Use global blocking semantics is enabled. Thisentails checking if there exists a vertex in L with the Blockpolicy defined for the EDS change type ch. In case sucha vertex exists, the algorithm returns an unchanged graph.On Line 7, we start traversing each vertex v of L. On Lines8–11, we check if the policy for v given ch is Block. If this isthe case, we can disregard this branch of the graph for thetraversal, which is why we remove all successors of v fromthe sorted list of vertices. Line 12 updates the dependenciesof a vertex. This is done by going through each dependencyin v.dependencies for a given vertex v and checking thatall of the involved columns are still present in the incom-ing edges. The reason for this is that some columns mighthave been removed from v’s incoming edges when traversingthe preceding vertices, such that v.dependencies refers tocolumns that no longer exist in v’s input.

As stated before, it is not sufficient to just look at verticesindependently when going through the output columns ofeach vertex. Lines 13–19 show the case for deleting columnswith no dependencies. As an example, consider a vertex vwhose preceding transformation is a Derived Column withthe input column c, which is used to derive a new columnd through the expression d = c + 42. Now, v will receivec and d as input columns, but if the EDS schema change(ch) is a deletion of c, not only will c not be available to v

Algorithm 1: Algorithm for propagating EDSschema changes to the graph

Name: Alter-graphInput: EDS-Change ch, Graph GOutput: Graph G

1 List L = topological-sort(G)2 if Use global blocking semantics is enabled then3 foreach Vertex v ∈ L do4 Policy p = lookup-policy(v.type, ch.type)5 if p = Block then6 return G

7 foreach Vertex v ∈ L (in topological order) do8 Policy p = lookup-policy(v.type, ch.type)9 if p = Block then

10 Remove all successors of v from L11 continue

12 Update v’s dependencies to not include deletedcolumns

13 foreach Column c ∈ v’s outgoing edges do14 if v.dependencies(c) = ∅ AND v.type 6= OLE

DB Source then15 if v is fully dependent on c then16 Delete v and v’s incoming and

outgoing edges from G17 Break inner loop

18 else19 Delete c from v’s corresponding

outgoing edges and dependencies

20 if v.type is OLE DB Destination AND v has noincoming edges then

21 Delete v

22 if v was not deleted AND ch affects v then23 G = alter(G, v, ch)

24 return G

anymore, d will also be deleted, as it is no longer possible toderive it from the Derived Column without c. The way tofind out that d is no longer computable is by seeing that d isdependent on ∅ (Line 14). This signifies that d was depen-dent on column(s) which have been deleted. The exceptionis an OLE DB Source, which is the only vertex for whichdependencies maps columns to ∅, as explained in Section3 (OLE DB Source is the only transformation allowing noincoming edges). Beginning on Line 15, it is considered ifit is possible to delete only the given column d or if it isnecessary to delete the entire vertex v. This narrows downto whether v is fully dependent on d or not. We say that avertex v is fully dependent on some column c, when v wouldbe rendered invalid if c was deleted. For instance, if v isan Aggregation and c is the only remaining column that isbeing used for aggregations, then v is fully dependent on c.The deletion of c would result in v.aggregations = ∅, whichis not a valid transformation. What qualifies a vertex asbeing fully dependent on a column is specific for each typeof transformation, for details see [3]. Lines 16–17 show thecase of deleting a whole vertex and all of its incoming andoutgoing edges, if they are no longer used. This iteration ofthe loop breaks because it would not make sense to continue

Table 2: Currently supported EDS changes.Deletion Rename

OLE DB Source 3 3

OLE DB Destination 3 3

Aggregate 3 3

Conditional Split 3 3

Data Conversion 7 7

Derived Column 3 3

Lookup 3 3

Sort 7 7

Union All 7 7

iterating over the output columns of a vertex that has beendeleted. The other case of only deleting the given columnis shown in Lines 18–19. Since a vertex of type OLE DBDestination does not have any outgoing edges, it is not con-sidered in the previous loop on Lines 13–19. However, westill want to delete the vertex if it is invalid, i.e., if it hasno incoming edge. This is performed on Lines 20–21. Af-terwards on Lines 22–23, if v was not deleted at an earlierpoint in the algorithm, then the corresponding propagationaction for the EDS schema change and transformation typeis invoked. Finally, the fully altered graph is returned. Re-call that the underlying SSIS Data Flow tasks gets updatedautomatically when the graph is updated.

5. IMPLEMENTATIONThis section describes the implementation of the proto-

type. MAIME was developed in C# for Microsoft SQLServer 2014 Developer Edition and with the SSIS Designeras the data integration tool. The SSIS Designer is part ofthe SQL Server Data Tools (v. 14.0.60203.0). The code-base for the implementation includes around 7,700 lines ofcode, 85 classes, and 410 members. The implementationis open source and is available from https://github.com/sajens/MAIME.

The advanced configurations, Allow deletion of transfor-mations and Use global blocking semantics are implemented.We did not implement Allow modification of expressionsyet. We focused our attention primarily on Allow dele-tion of transformations and Use global blocking semantics,since they seemed to be the most impactful configurationsfor MAIME. Currently, Allow deletion of transformations isalways enabled. The Propagate and Block policies are imple-mented as described previously. Table 2 shows which trans-formations have been implemented with respect to deletionand renaming of EDS columns in the current prototype.

We now consider how the graph is represented internallyin MAIME. Our graph is implemented as a layer on top ofSSIS. Therefore, our graph constructs have references to thecorresponding SSIS constructs. To be more exact, each ver-tex refers to a SSIS component (transformation), each edgerefers to a SSIS path, and each column has a correspondingreference to a so-called SSIS IDTS object. The SSIS IDTSobject covers both input and output columns in SSIS. Byhaving these references, we can easily propagate changes tothe underlying ETL process during execution of the graphalteration algorithm. For the graph, we have implementedthe classes Graph, Vertex, Edge, Column, and a class ex-tending Vertex for each supported transformation such thatspecific dependencies and properties can be represented.

In order to instantiate the graph, we have to translate thecomponents from SSIS1. Initially, a SSIS package is loadedby parsing its .dtsx file conforming to XML specification.Then, we extract its SSIS Data Flow tasks. Note that everyData Flow task is stored as a graph in a .dtsx file. Con-struction of a MAIME graph is done in two iterations. Dur-ing the first iteration, we extract one Data Flow task andgo through its components to create corresponding verticeswith references to the Data Flow task components. Withthe set of vertices V and references to the SSIS components,we can extract information for each vertex such as v.nameand v.type. Afterwards, every SSIS path is translated intoan edge. This gives us the set of edges E of our graph. It isimportant to note that columns are not stored in the edgesin this iteration, since they do not exist on SSIS paths. Af-ter the first iteration is complete, we have created the basicstructure of the graph of MAIME. In the second iteration,we deduce dependencies, assign columns to edges, and createtransformation specific properties.

6. EVALUATIONWe now evaluate the efficiency of MAIME. In the evalua-

tion, we compare how much time and how many user inputsare required to resolve a series of EDS schema changes whenusing MAIME and when repairing the ETL flow manually.A user input is defined to be a mouse click or a keystrokedone by the user (both are recorded separately). We dis-cussed and validated this approach with practitioners in fieldof ETL development.

We consider three SSIS packages and each has a singleData Flow task. For each considered SSIS package, bothMAIME and the manual work end up having the same endresult, i.e. the same state of the maintained SSIS package.Only the process of achieving that end result differs. TheEDS schema changes used for the evaluation include deletionand renaming.

For the manual approach, the user performing the evalu-ation (one of the authors) is given the EDS schema changesand two versions of the ETL process: One in its initial state,and another in the desired result state. The user then per-forms maintenance of each package three times using VisualStudio 2015 Community edition and SQL Server Data Tools.The first evaluation gives an indication of how long it takesto maintain an ETL process without knowing every step tomaintain the ETL process. The second and third attemptare performed in order to provide a best-case scenario, sincethe user learns the quickest way to maintain the ETL pro-cess through repetition. The time of the evaluation startswhen the user begins repairing the ETL process after havingseen the given information (i.e., EDS schema changes andthe two versions of the ETL process). During the test, theduration and amount of user inputs used are recorded usinga software tool.

When considering MAIME, the user interacts with MAIMEto (1) accept the configurations, currently loaded ETL pro-cesses, and connected EDSs, and (2) start the maintenanceprocess. The EDS schema change configurations of MAIMEfor all evaluations are all set to propagate. For the advancedconfigurations, Allow deletion of transformations is enabled,Allow modification of expressions is disabled, and Use globalblocking semantics is disabled. The time of the evaluation

1msdn.microsoft.com/en-us/library/ms403344.aspx

starts when the user begins to use MAIME and ends whenthe SSIS package has been repaired. MAIME is also used torepair the package three times and the duration and amountof user inputs are again recorded using a software tool.

We acknowledge that the user knows MAIME very welland thus easily can use it. That is why the user also is giventhe desired resulting SSIS package for the manual approachsuch that he does not have to think about what needs to bechanged. This is thus an overly optimistic way to measurethe needed time for the manual approach (the amounts ofneeded clicks and keystrokes are not affected, though).

We now elaborate on one of our test cases and a given listof EDS schema changes. In test case 1, we consider the sce-nario from Example 1 and assume the following changes tothe EDS: (1) Age is renamed to RenamedAge in the Persontable and (2) TotalAmount is deleted from the Sale table.The state of the graph after it has been maintained is shownin Figure 6 in the SSIS designer tool, and with MAIME’sgraph in Figure 7. The latter shows that Age is success-fully renamed on all edges. The deletion of TotalAmount isslightly more complicated since the condition TotalAmount> 10000 in the Conditional Split involves the column. Thiscondition is no longer valid which results in the removal ofthe outgoing edge containing it. We therefore delete OLEDB Destination, as it no longer has any incoming edges.Derived Column derives the column AmountTimes10 fromTotalAmount and can therefore no longer be derived. Wedelete this derivation but still retain our Derived Columntransformation since it can exist without doing any deriva-tions. Another possibility would be to let MAIME delete theDerived Column and connect the Lookup and ConditionalSplit transformations. The Aggregate transformation takesthe average of AmountTimes10, which no longer exists andthis aggregation is therefore also deleted. OLE DB Desti-nation 1 no longer loads the aggregated AvgAmount fromthe Aggregate transformation into the DW. Both TotalAm-ount, AmountTimes10, and AvgAmount are deleted fromthe edges.

Figure 6: The Data Flow task of the SSIS packagefor test case 1 after the EDS schema changes.

Test cases 2 and 3 are not shown here for space reasons,but only briefly described. In test case 2, TotalAmount isdeleted, and both Address and Person.ID are renamed. Intest case 3, Name is deleted and Age is renamed.

For each test case, the used time is shown in Table 3 andthe amounts of user inputs are shown in Table 4. For time-consumption, MAIME took on average 4 seconds across all3 test cases while manually resolving changes took 39.3 sec-onds on average across all 3 cases for the third attempt. Thismeans MAIME was on average of 9.8 times faster for resolv-

Columns = {(1, ID, int),(2, Name, string),(3, RenamedAge, int)}

Name OLE DB Source Database = SourceDB Table = Person Columns = { ID, Name, RenamedAge}Dependencies = {ID ØName ØRenamedAge Ø }


1 2 3

Name Lookup Database = SourceDB Table = Sale Columns = { ID, CustomerID, StoreID}OutputColumns = {}Joins = {(ID, CustomerID)}Dependencies = {ID {ID}Name {Name}RenamedAge {RenamedAge}}

Name = Derived Column Derivations = {}Dependencies = {ID {ID}Name {Name}RenamedAge {RenamedAge}}


4

Name = Conditional Split Conditions = { RenamedAge > 40", 1, Aggregate)}Dependencies = {ID {ID}Name {Name}RenamedAge {RenamedAge}}


67

Name = Aggregate Aggregations = { (GROUP BY, RenamedAge, Age, OLE DB Destination 1)}Dependencies = {Age {RenamedAge}}

Columns = {(4, Age, int)}

Name = OLE DB Destination 1 Database = TargetDW Table = SalesData Columns = {Age}

Figure 7: The MAIME graph for the Data Flow taskin Figure 6.

Table 3: Time needed for handling EDS schemachanges for the three test cases.

Time Elapsed MAIME ManualTest Case 1 4 sec, 4 sec, 4 sec 187 sec, 159 sec, 59 secTest Case 2 4 sec, 4 sec, 4 sec 154 sec, 60 sec, 49 secTest Case 3 4 sec, 4 sec, 4 sec 23 sec, 13 sec, 10 sec

ing EDS schema changes. For user input, MAIME requiredon average 4 user inputs, while manually resolving changesrequired 38 user inputs on average for the third attemptacross all 3 cases. This means MAIME on average required9.5 times less inputs for resolving EDS schema changes.

For the manual work, it took significantly more time thefirst time the user had to maintain the ETL process. Theuser did, however, get progressively quicker at maintainingthe ETL processes for the second and third attempt, withthe best result being from the third attempt. The number ofboth mouse clicks and keystrokes showed similar results. Forthese evaluations MAIME is thus able to significantly reducethe amount of time and user input needed for maintainingETL processes.

In a real life scenario, a company would have to maintaina multitude of ETL processes, which for each could takeon average 39.3 seconds to repair as our evaluation showed.However, the timings shown for MAIME in Table 3 (4 sec-onds) include the time to first load the ETL processes, detectEDS schema changes, and the time the user spent on click-ing buttons in the GUI. If there were many ETL processes,it would still be enough to do this once. The reparation al-gorithm of MAIME uses less than a second in all the cases,and we thus argue that the evaluation setup actually givesthe manual process an advantage.

7. RELATED WORKRelated work exists on the topic of maintaining ETL pro-

cesses when the schema of EDSs change. The frameworkHecataeus by Papastefanatos et al. [7, 8, 9] analyzes ETLprocesses by detecting changes to the schema of the EDSs

Table 4: User inputs needed for handling EDSschema changes for the three test cases.User Inputs MAIME Manual

Test Case 1Keystrokes: 0, 0, 0Mouse clicks: 4, 4, 4

Keystrokes: 23, 15, 12Mouse clicks: 88, 85, 38





(called evolution events) and proposes changes to the ETLprocesses based on defined policies. The Hecataeus frame-work abstracts ETL processes as SQL queries and viewswhich are used in a graph to represent the activities. An ac-tivity resembles an ETL transformation. Hecataeus’s graphis captured by an ETL Summary which can have multiplesubgraphs for each activity. Each activity can further bebroken down, as it includes nodes or entire subgraphs forrelations, SQL queries, conditions, and uses views, e.g., forinput and output. The types of evolution events taken intoaccount are addition, deletion, and modification. An ad-ministrator can annotate nodes and edges with policies foreach type of evolution event, while in MAIME policies areprovided for each EDS change type and not specifically foreach node or edge. The approach taken by MAIME wouldbe preferable if the administrator had to repair a lot of ETLprocesses. The three Hecataeus policies dictate how ETLprocesses should be adjusted when an evolution event oc-curs: (1) Propagate readjusts the graph to reflect the newsemantics according to the evolution event throughout therest of the graph, (2) Block retains the old semantics, and(3) Prompt asks the administrator to choose Propagate orBlock. Both Propagate and Prompt are similar in MAIME,whereas their Block attempts to retain the semantics. Anextension of Hecataeus has been made for what-if analysis ofevolutions of relations, views, or queries in a data-intensiveecosystem [6]. While MAIME models columns on edges,Hecataeus models input and output schemata as subgraphs.

Another framework is E-ETL [14, 15] by Wojciechowskiwhich is also able to semi-automatically adapt ETL pro-cesses to EDS schema changes. In order to adapt the ETLprocesses, there are different methods which define how repa-rations are propagated. E-ETL has the same three policiesas Hecataeus, with the exception of the Block policy. InE-ETL the Block policy ignores the EDS schema changeand does not attempt to modify the graph. The E-ETLframework has multiple ways to handle an EDS schemachange. These include Defined rules in which the admin-istrator can define rules himself for nodes and edges; Stan-dard rules which are the default rules used if the admin-istrator did not define anything; and Alternative scenarioswhere case-based reasoning is used to adjust the ETL pro-cess based on solutions to similar problems experienced pre-viously. How these algorithms are used together is not speci-fied. Like Hecataeus, E-ETL models ETL processes throughSQL queries.

Related work also exists on how to conceptually modelETL processes using transformations that are commonlyused in ETL processes. Trujillo et al. [13] do this by defin-ing a small set of transformations modeled through UMLdiagrams. Examples of the mechanism include Aggrega-tion, Conversion, and Join. This is in contrast to other

frameworks, such as Hecataeus and E-ETL, where ETL pro-cesses are described by SQL queries and views, and repre-sented with graphs. Using transformations rather than SQLqueries can make conceptual modeling simpler and moremaintainable as each transformation has a clear responsi-bility and provides a higher level of abstraction comparedto SQL queries, but at the cost of expressiveness. Trujilloet al. provide a set of general ETL processes that are notspecific to any platform, while our transformations are spe-cific to SSIS. Furthermore, instead of using UML diagramswe use property graphs to more closely relate to the internalgraphs of SSIS. In [11], conceptual models are specified witha kind of diagrams which then are mapped to logical modelswhere graphs also are used.

Business Intelligence Markup Language (BIML) [1] is anXML dialect that can be used to generate SSIS packages. Itcan be used to create templates and reusable code snippetsto make the creation of SSIS packages easier and less time-consuming. BIML does, however, not handle changes, butcan rather re-generate packages when a change occurs.

8. CONCLUSION AND FUTURE WORKETL processes can be complex and time-consuming to re-

pair manually. We presented MAIME as a tool to reducethe amount of errors and time spent on maintaining ETLprocesses. To accomplish this task, we introduced and im-plemented a graph model as a layer on top of SSIS DataFlow tasks, which simplifies the handling of EDS schemachanges.

As we have seen in the evaluation, MAIME required 9.5times less input from the user and was 9.8 times faster com-pared to doing it manually with the SSIS Designer tool.MAIME can thus ease the burden of maintaining ETL pro-cesses.

There are a number of possible directions for future work.The paper presents an adaptive way of handling source datachanges in ETL programs. One direction for future work isto make MAIME more robust against the variance of sourcetypes and change types. The adaptivity very often dependson manual decisions made by the ETL administrator. Thus,an interesting extension of this work is to model or formalizethe pattern of the manual decisions and predict the next de-cisions such that the system execution process involves lessmanual interruptions. Since the system starts to make auto-matic changes to ETL programs, it is very natural that theETL administrators considers to “roll-back” the execution ofcertain groups of ETL programs to the previous state. Howto define the transaction process in such an ETL programand how to ensure the possibility of the roll-back action isanother direction to explore.

We have so far disregarded propagating changes to theDW, e.g., adding a column to the relevant DW schema.However, an administrator might want to do proper adjust-ments to the DW in order to capture the correct semanticsof an EDS schema change. For example, when a column wasinserted in an EDS, it might be reasonable to insert a corre-sponding column into the DW. This is another direction toinvestigate. It could also be interesting to investigate howto maintain ETL programs when the target DW is changed.

Another direction for future work is to extend the currentprototype. Currently, we extract only one SSIS Data Flowtask from the current SSIS package. An improvement would

be to extract all SSIS Data Flow tasks from each SSIS pack-age. Furthermore, support for more SSIS transformationscould be implemented as well as support for SSIS Con-trol Flow tasks. For example, it would be interesting tohave support for more kinds of data sources, such as XML(where a schema can be explicitly given) and a CSV (wherea schema typically has to be inferred). We note that a fewlocal software vendors and IT service providers have been inthe progress of adopting and extending MAIME into theirproduct and service offerings.

9. ADDITIONAL AUTHORSHarry Xuegang Huang (Danmarks Nationalbank, email:

[email protected]) and Christian Thomsen (Dept. ofComp. Science, Aalborg University, email: [email protected]).

10. REFERENCES[1] Business Intelligence Markup Language.

https://www.varigence.com/biml (Last accessed2016-10-31).

[2] SSIS Designer. https://msdn.microsoft.com/en-us/library/ms137973(v=sql.120).aspx (Last accessed2016-10-31).

[3] D. Butkevicius et al. MAIME - Maintenance Managerfor ETL. Student project d802f16, Dept of Comp.Science, Aalborg University, August 2016.https://github.com/sajens/MAIME/blob/master/Technical-Report.pdf.

[4] R. Kimball and M. Ross. The Data WarehouseToolkit. John Wiley & Sons, 2011.

[5] B. Knight et al. Professional Microsoft SQL Server2012 Integration Services. John Wiley & Sons, 2012.

[6] P. Manousis, P. Vassiliadis, and G. Papastefanatos.Impact Analysis and Policy-Conforming Rewriting ofEvolving Data-Intensive Ecosystems. Journal on DataSemantics, 4(4), 2015.

[7] G. Papastefanatos et al. Rule-Based Management ofSchema Changes at ETL Sources. ADBIS, 2010.

[8] G. Papastefanatos et al. What-If Analysis for DataWarehouse Evolution. DaWaK, 2007.

[9] G. Papastefanatos et al. Policy-RegulatedManagement of ETL Evolution. Journal on DataSemantics XIII, 2009.

[10] M. A. Rodriguez and P. Neubauer. Constructions fromdots and lines. Bulletin of the American Society forInformation Science and Technology, 2010.

[11] A. Simitsis. Mapping Conceptual to Logical Modelsfor ETL Processes. DOLAP, 2005.

[12] E. Thoo and M. A. Beyer. Gartner’s Magic Quadrantfor Data Integration Tools. 2014.

[13] J. Trujillo and S. Lujan-Mora. A UML BasedApproach for Modeling ETL Processes in DataWarehouses. ER, 2003.

[14] A. Wojciechowski. E-ETL: Framework for ManagingEvolving ETL Workflows. Foundations of Computingand Decision Sciences, 38(2), 2013.

[15] A. Wojciechowski. E-ETL Framework: ETL ProcessReparation Algorithms Using Case-Based Reasoning.ADBIS Short Papers and Workshops, 2015.

maime: a maintenance manager for etl processesceur-ws.org/vol-1810/dolap_paper_08.pdfmaime: a...

Documents