1 program comprehension and software migration strategies hausi a. müller university of victoria...

11

Program Comprehension Program Comprehension and Software Migration and Software Migration

StrategiesStrategies

Hausi A. MüllerHausi A. Müller

University of VictoriaUniversity of Victoria

IWPC-2000IWPC-2000

Limerick, Ireland, June 11, 2000Limerick, Ireland, June 11, 2000

ICSE 2000 Roadmap ICSE 2000 Roadmap 22

OutlineOutline

Reengineering categoriesReengineering categories Comprehension strategiesComprehension strategies Migration strategiesMigration strategies Language migrationLanguage migration Program comprehension educationProgram comprehension education Mt. St. Helens TheoryMt. St. Helens Theory Key research pointersKey research pointers ConclusionsConclusions


Research SupportResearch Support


The Horseshoe ModelThe Horseshoe Modelof Software Migrationof Software Migration

Old system New system

Abstract system


Reengineering Reengineering CategoriesCategories

Automatic restructuringAutomatic restructuring Automatic transformationAutomatic transformation Semi-automatic transformationSemi-automatic transformation Design recovery and reimplementationDesign recovery and reimplementation Code reverse engineering and forward Code reverse engineering and forward

engineeringengineering Data reverse engineering and schema Data reverse engineering and schema

migrationmigration Migration of legacy systems to modern Migration of legacy systems to modern

platformsplatforms


The Horseshoe ModelThe Horseshoe Model

Existing system New system

Abstract system

Semi-automaticSemi-automatic

AutomaticAutomatic

ComponentsComponents MiddlewareMiddleware


Reengineering Reengineering Categories...Categories...

Automatic restructuringAutomatic restructuring• to obtain more readable source codeto obtain more readable source code• enforce coding standardsenforce coding standards

Automatic transformationAutomatic transformation• to obtain better source codeto obtain better source code• HTML’izing of source codeHTML’izing of source code• simplify control flow (e.g., dead code, goto’s)simplify control flow (e.g., dead code, goto’s)• refactoring and remodularizeingrefactoring and remodularizeing• Y2K remediationY2K remediation


Reengineering Reengineering Categories...Categories...

Semi-automatic transformationSemi-automatic transformation• to obtain better engineered system (e.g., to obtain better engineered system (e.g.,

rearchitect code and data)rearchitect code and data)• semi-automatic construction of structural, semi-automatic construction of structural,

functional, and behavioral abstractionsfunctional, and behavioral abstractions• re-architecting or re-implementing the re-architecting or re-implementing the

subject system from these abstractionssubject system from these abstractions


Design RecoveryDesign RecoveryLevels of AbstractionsLevels of Abstractions

ApplicationApplication• Concepts, business rules, policiesConcepts, business rules, policies

FunctionFunction• Logical and functional specifications,Logical and functional specifications,

non-functional requirementsnon-functional requirements StructureStructure• Data and control flow, dependency graphsData and control flow, dependency graphs• Structure and subsystem chartsStructure and subsystem charts• ArchitecturesArchitectures

ImplementationImplementation• AST’s, symbol tables, source textAST’s, symbol tables, source text


Synthesizing ConceptsSynthesizing Concepts

Build multiple hierarchical mental modelsBuild multiple hierarchical mental models Subsystems based on SE principlesSubsystems based on SE principles• classes, modules, directories, cohesion,classes, modules, directories, cohesion,

data & control flows, slicesdata & control flows, slices

Design and change patternsDesign and change patterns Business and technology modelsBusiness and technology models Function, system, and application architecturesFunction, system, and application architectures Common services and infrastructureCommon services and infrastructure


Modeling Mental Modeling Mental ModelsModels

The Ubiquitous Graph The Ubiquitous Graph ModelModel

Subsystem

ClassificationTyped nodes and arcs

Generalization arcs

Subsystem

Aggregation arcs

Composite arcComposite node


Program Program Comprehension Comprehension

TechnologyTechnology Program understanding technologyProgram understanding technology• Cognitive modelsCognitive models• Levels of abstractionLevels of abstraction• Synthesizing conceptsSynthesizing concepts• Filtering informationFiltering information• Slicing and dicingSlicing and dicing

Comprehension environmentComprehension environment• Parsers and lightweight extractorsParsers and lightweight extractors• Repository and conceptual modelingRepository and conceptual modeling• Visualization engines (graph and web based)Visualization engines (graph and web based)


The Big-Bang The Big-Bang Comprehension ProblemComprehension Problem

What can we do during evolution to What can we do during evolution to ease future understanding and migration ease future understanding and migration of information systems?of information systems?

We know the knowledge we need butWe know the knowledge we need butit is difficult to obtain from scratchit is difficult to obtain from scratch

““Big-bang” comprehension when the Big-bang” comprehension when the system becomes “critical” is high-risksystem becomes “critical” is high-risk

Analysis paralysisAnalysis paralysis


The Understanding GapThe Understanding Gap

needed overall understanding

t

useful, known overall understanding

[Wong99][Wong99]

t1t1 t2t2


Continuous Program Continuous Program ComprehensionComprehension

Apply program understanding Apply program understanding continuously and incrementally during continuously and incrementally during evolution of the software systemevolution of the software system

Use software reverse engineering toUse software reverse engineering tore-document existing softwarere-document existing software

Insert reverse engineering techniques Insert reverse engineering techniques into development [Wong99]into development [Wong99]

Symbiosis: models and code [Jackson00]Symbiosis: models and code [Jackson00]


Evaluating Reverse Evaluating Reverse Engineering ToolsEngineering Tools

The purpose of most reverse engineering The purpose of most reverse engineering tools is to increase the understanding an tools is to increase the understanding an engineer has of the subject systemengineer has of the subject system

No agreed-upon definition or test of No agreed-upon definition or test of understandingunderstanding

Several types of empirical studies that Several types of empirical studies that are appropriate for studying the benefits are appropriate for studying the benefits of reverse engineering toolsof reverse engineering tools


Program Understanding Program Understanding ThesesTheses

An Emerging DisciplineAn Emerging Discipline Domain retargetable reverse engineering Domain retargetable reverse engineering

[Tilley95][Tilley95] Cognitive design elements for software Cognitive design elements for software

exploration tools [Storey98]exploration tools [Storey98] Continuous understanding ReverseContinuous understanding Reverse

Engineering Notebook [Wong99]Engineering Notebook [Wong99] Integrating static and dynamic reverse Integrating static and dynamic reverse

engineering models [Systa2000]engineering models [Systa2000] Architectural Component Detection forArchitectural Component Detection for

Program Understanding [Koschke2000]Program Understanding [Koschke2000]


OutlineOutline



Migration ThesesMigration Theses

Management of uncertainty and inconsistency Management of uncertainty and inconsistency in database reengineering [Jahnke99]in database reengineering [Jahnke99]

Integration and migration of information Integration and migration of information systems to object-oriented platforms systems to object-oriented platforms [Koelsch99][Koelsch99]

Migrating C++ to Java [Agrawal99, Wen2000]Migrating C++ to Java [Agrawal99, Wen2000] An Environment for Migrating C to Java An Environment for Migrating C to Java

[Martin2000][Martin2000]


Migration ObjectivesMigration Objectives Evolving Business Requirements Evolving Business Requirements

Adapt to e-commerce platformAdapt to e-commerce platform Adapt to web technologyAdapt to web technology Reduce time to marketReduce time to market Support new business rulesSupport new business rules Allow customizable billingAllow customizable billing Adapt to evolving tax lawsAdapt to evolving tax laws Reengineer business processesReengineer business processes


Migration Objectives …Migration Objectives … Software Evolution Requirements Software Evolution Requirements

Higher productivityHigher productivity Lower maintenance costsLower maintenance costs Move to object-oriented platformsMove to object-oriented platforms Inject component technologyInject component technology Adapt to modern data exchange Adapt to modern data exchange

technologytechnology Leverage modern methods and toolsLeverage modern methods and tools


Migration Objectives …Migration Objectives … Software Architecture Software Architecture

RequirementsRequirements

Move to network-centric platformsMove to network-centric platforms Integrate cooperative information systemsIntegrate cooperative information systems Leverage centralized repositoriesLeverage centralized repositories Move from hierarchical to relational dbMove from hierarchical to relational db Take advantage of web user interfacesTake advantage of web user interfaces Provide interoperability via buses and Provide interoperability via buses and

gateways among applicationsgateways among applications Move to client-server architecturesMove to client-server architectures


Common Requirements Common Requirements MigrationMigration

Ensure continuous, safe, reliable, robust, ready access Ensure continuous, safe, reliable, robust, ready access to mission-critical functions and informationto mission-critical functions and information• Migrate in placeMigrate in place

Minimize migration riskMinimize migration risk• Reduce migration complexityReduce migration complexity• Make as few changes as possible in both code & Make as few changes as possible in both code &

datadata• Alter the legacy code to facilitate and ease migrationAlter the legacy code to facilitate and ease migration• Concentrate on the most important current and future Concentrate on the most important current and future

requirementsrequirements


Common Migration Common Migration Requirements ...Requirements ...

Minimize impact onMinimize impact on• usersusers• applicationsapplications• databasesdatabases• operationoperation

Maximize benefits of modern technologyMaximize benefits of modern technology• user interfaces, dbs, middleware, COTSuser interfaces, dbs, middleware, COTS• automation, toolsautomation, tools


Dimensions of Dimensions of MigrationMigration

Methods and ToolsMethods and Tools

AutomationAutomation

ScaleScale

DomainDomain

automaticautomatic

manualmanual

10K10K 10M10M

genericgeneric

specificspecific

User involvementUser involvement


Resistance to ChangeResistance to Change

Are some systems more difficult to Are some systems more difficult to change, evolve, reengineer than change, evolve, reengineer than others?others?

Can we define a measure resistance Can we define a measure resistance based on business value, existing based on business value, existing technology, new technology, evolution technology, new technology, evolution pace?pace?

We need empirical studies ...We need empirical studies ...


Separable TiersSeparable Tiers

Decompose legacy system into three Decompose legacy system into three layers or application tierslayers or application tiers• Presentation (interfaces: user and APIs)Presentation (interfaces: user and APIs)• Processing (application code, functions, Processing (application code, functions,

business rules, policies)business rules, policies)• Data services (database)Data services (database)

Promotes interoperability, reuse, flexibility, Promotes interoperability, reuse, flexibility, distribution, separate evolution pathsdistribution, separate evolution paths


Application LayersApplication Layers

Processing ObjectsProcessing Objects InfrastructureInfrastructure

User ObjectsUser Objects

Data ObjectsData Objects


Classification of LIS Classification of LIS ArchitecturesArchitectures

DecomposableDecomposable• Separation of concernsSeparation of concerns• Interfaces, applications, db services are Interfaces, applications, db services are

distinct componentsdistinct components• Functional decompositionFunctional decomposition• Ideal for migrationIdeal for migration

There is nothing more difficult to arrange, more doubtful of success, and more dangerous to carry through than initiating changes.

—N. Machiavelli


Classification of IS Classification of IS Architectures ...Architectures ...

SemidecomposableSemidecomposable• Applications and db services are not Applications and db services are not

readily separablereadily separable• System is not easily decomposableSystem is not easily decomposable

NondecomposableNondecomposable• No functional components are separableNo functional components are separable• Users directly interact with individual Users directly interact with individual

modulesmodules [BS95][BS95]


Migration StrategiesMigration Strategies IgnoreIgnore• retire, phase out, let failretire, phase out, let fail

Replace with COTS applicationsReplace with COTS applications Cold turkeyCold turkey• rewrite from scratchrewrite from scratch• high riskhigh risk

Integrate and access in placeIntegrate and access in place• integrate future apps into legacy apps integrate future apps into legacy apps

without modifying legacy appswithout modifying legacy apps• IS-GTP [Koelsch99]IS-GTP [Koelsch99]


Data WarehousingData Warehousing

Data is needed for several distinct Data is needed for several distinct purposespurposes• on-line transaction processing (access in place)on-line transaction processing (access in place)• data analysis for decision support applications data analysis for decision support applications

(extraction of data into an application specific (extraction of data into an application specific repository)repository)

Creates duplicate dataCreates duplicate data Popular approachPopular approach


Gradual Migration or Gradual Migration or “Chicken Little”“Chicken Little”

Rearchitect and transition the Rearchitect and transition the applications incrementallyapplications incrementally

Replace LIS with target applicationReplace LIS with target application Language migrationLanguage migration Schema and data migrationSchema and data migration User interface migrationUser interface migration GTE [BrSt95]GTE [BrSt95]


Chicken Little ...Chicken Little ...

The intent is to phase out legacy The intent is to phase out legacy applications over timeapplications over time

In place access is not economical in the In place access is not economical in the long runlong run

More effective, less risky than cold turkeyMore effective, less risky than cold turkey Allows for independent user interface and Allows for independent user interface and

database evolutiondatabase evolution IncrementalIncremental


Chicken Little ...Chicken Little ...

Legacy and target applications must Legacy and target applications must coexist during migrationcoexist during migration

A gateway to isolate the migration steps A gateway to isolate the migration steps so that the end users do not know if the so that the end users do not know if the info needed is being retrieved from the info needed is being retrieved from the legacy or target systemlegacy or target system

Development of gateways is difficult and Development of gateways is difficult and costlycostly


Opportunistic Migration Opportunistic Migration MethodMethod

Combination of forward and reverse Combination of forward and reverse migration strategiesmigration strategies

Forward or reverse migration path perForward or reverse migration path per• operationoperation• applicationapplication• interfaceinterface• databasedatabase• sitesite• useruser

More complex gateways are neededMore complex gateways are needed


Migration Research Migration Research MethodMethod

Perform a concrete case study with an Perform a concrete case study with an industrial software systemindustrial software system

Investigate methods and tools to Investigate methods and tools to automate the process adopted in the automate the process adopted in the case studycase study

Conduct user experiments to improve the Conduct user experiments to improve the effectiveness of the developed methods effectiveness of the developed methods and toolsand tools

Investigate tool adoption problemsInvestigate tool adoption problems


Subject system is a 300 KLOC legacy Subject system is a 300 KLOC legacy software system of highly optimizedsoftware system of highly optimizedcode written in PL/IXcode written in PL/IX

Can the system incrementally be Can the system incrementally be translated to C++?translated to C++?• Transliteration versus object-oriented designTransliteration versus object-oriented design

Develop tools which semi-automate the Develop tools which semi-automate the translation process to C++translation process to C++

The translated code must perform as well The translated code must perform as well as the original codeas the original code

Language Migration—A Language Migration—A Case StudyCase Study


First migration and integration effort was First migration and integration effort was completed by hand by an expert [Uhl97]completed by hand by an expert [Uhl97]

10 person-weeks to migrate 7.8 KLOC10 person-weeks to migrate 7.8 KLOC Successfully passed all regression testsSuccessfully passed all regression tests Built C++ and Fortran compilers with itBuilt C++ and Fortran compilers with it It works …It works …

… but migrated C++ code was 50% … but migrated C++ code was 50% slower than original PL/IX codeslower than original PL/IX code

Manual MigrationManual Migration


Expert identified performance bottlenecksExpert identified performance bottlenecks Hand-optimized migrated codeHand-optimized migrated code Optimized version performed better than Optimized version performed better than

the original version [Martin98]the original version [Martin98]• Up to 20% better than the original codeUp to 20% better than the original code• Now IBM was interested …Now IBM was interested …

ResultsResults• Correct, efficientCorrect, efficient• Translation, integration, optimization heuristicsTranslation, integration, optimization heuristics• Incremental processIncremental process

Performance EvaluationPerformance Evaluation


Can the translation, integration, and Can the translation, integration, and optimization heuristics discovered by optimization heuristics discovered by experts be integrated into anexperts be integrated into anautomated tool?automated tool?

How would it affect the performance?How would it affect the performance? What existing tools could be leveragedWhat existing tools could be leveraged

to build such a tool?to build such a tool? SolutionSolution• Use Software Refinery, Reasoning SystemsUse Software Refinery, Reasoning Systems

AutomationAutomation


Transformation ProcessTransformation Process

Transform PLI/IX artifacts to their Transform PLI/IX artifacts to their corresponding C++ artifactscorresponding C++ artifacts

Generate support C++ libraries (macros for Generate support C++ libraries (macros for reference components; class definitions for reference components; class definitions for key data structures)key data structures)

Generate C++ source code that is Generate C++ source code that is structurally and behaviorally similar to the structurally and behaviorally similar to the legacy source codelegacy source code

CASCON98 Best Paper [Kontogiannis98]CASCON98 Best Paper [Kontogiannis98]


Semi-automatic transformation ofSemi-automatic transformation oflarge volume of code is feasiblelarge volume of code is feasible

Migrated code suffers no deteriorationMigrated code suffers no deteriorationin performancein performance

Incremental migration process feasibleIncremental migration process feasible Technique readily applicable to other Technique readily applicable to other

imperative languagesimperative languages Tool reduces migration effort by a factor Tool reduces migration effort by a factor

of 10 over manual migrationof 10 over manual migration CTAS—C++ to Java [Jackson2000]CTAS—C++ to Java [Jackson2000]

Results, Morale & Results, Morale & Lessons LearnedLessons Learned


OutlineOutline



How many teach 4th year or graduate How many teach 4th year or graduate courses in software evolution, program courses in software evolution, program understanding, comprehension, understanding, comprehension, reverse engineering, reengineering?reverse engineering, reengineering?

How many teach program How many teach program understanding or program reading in understanding or program reading in 1st year?1st year?

Teaching program Teaching program understandingunderstanding


Mary Shaw, Software Engineering Mary Shaw, Software Engineering Education—A Roadmap; in The Future Education—A Roadmap; in The Future of Software Engineering, ICSE 2000of Software Engineering, ICSE 2000

1.1. Discriminate among different software Discriminate among different software development rolesdevelopment roles

4.4. Integrate an engineering point of view Integrate an engineering point of view into CS and IS undergraduate curriculainto CS and IS undergraduate curricula

6.6. Exploit our own technology in support Exploit our own technology in support of educationof education

Challenges and Challenges and AspirationsAspirations


Available knowledge about software Available knowledge about software exceeds what any one person can knowexceeds what any one person can know

Specializing rolesSpecializing roles Comprehension versus coding skillsComprehension versus coding skills Developing the role of a reverse Developing the role of a reverse

engineer, program comprehenderengineer, program comprehender Software inspection expertSoftware inspection expert

Discriminate among different Discriminate among different software development rolessoftware development roles


Study good examples of software Study good examples of software systems and develop program systems and develop program understanding skillsunderstanding skills

Teach back-of-the-envelope estimation Teach back-of-the-envelope estimation using reverse engineering technologyusing reverse engineering technology

Teach students how to investigate non-Teach students how to investigate non-functional requirements using program functional requirements using program comprehension technologycomprehension technology

Integrate an engineering point of Integrate an engineering point of view into undergraduate curriculaview into undergraduate curricula


Employ software exploration and Employ software exploration and reverse engineering tools in 1st yearreverse engineering tools in 1st year

Integrated environments such asIntegrated environments such asVA Java or J++ do not provide facilities VA Java or J++ do not provide facilities to explore and record mental modelsto explore and record mental models

Familiarize students with software Familiarize students with software exploration and conceptual modeling exploration and conceptual modeling toolstools

Restructure curricula to teach both fresh Restructure curricula to teach both fresh creation and evolutionary changecreation and evolutionary change

Exploit our own technology Exploit our own technology in support of educationin support of education


May 18, 1980May 18, 1980Mt. St. HelensMt. St. Helensself-destructed, setting off the biggest self-destructed, setting off the biggest landslide in recorded history and losing landslide in recorded history and losing 400 meters of its crown400 meters of its crown

Forests and meadows, and mountain Forests and meadows, and mountain streams were transformed into an ash-streams were transformed into an ash-gray wastelandgray wasteland

Ecologists dogma—nature recreates Ecologists dogma—nature recreates ecosystems in a predictable fashionecosystems in a predictable fashion

Mt. St. Helens Mt. St. Helens TheoryTheory


A decade later evenA decade later evenon the most sterile ofon the most sterile oflandscapes brave little vegetative landscapes brave little vegetative beachheads are formedbeachheads are formed

The unpredictability of recolonization The unpredictability of recolonization and the pivotal importance of chance in and the pivotal importance of chance in rebuilding of biological communitiesrebuilding of biological communities

Wildflower gardens, which are mixes of Wildflower gardens, which are mixes of lupine, Indian paintbrush, pearly lupine, Indian paintbrush, pearly everlasting, and fireweed, are emergingeverlasting, and fireweed, are emerging

A decade A decade laterlater


Is program comprehension research Is program comprehension research becoming too predictable?becoming too predictable?

Do we need a cataclysmic event to Do we need a cataclysmic event to rejuvenate comprehension research?rejuvenate comprehension research?

There are many vegetative beachheads There are many vegetative beachheads in the communityin the community

But they tend to gravitate towards But they tend to gravitate towards established research and toolsestablished research and tools

Particularly the tools arena needs new Particularly the tools arena needs new beachheads beachheads

Encourage island-Encourage island-driven researchdriven research


OutlineOutline



Key Research PointersKey Research Pointers

Investigate infrastructure, methods,Investigate infrastructure, methods,and tools for continuous program and tools for continuous program understanding to support the entire understanding to support the entire evolution of a software system from the evolution of a software system from the early design stages to the long-term early design stages to the long-term legacy stageslegacy stages• Reverse engineering notebookReverse engineering notebook


Key Research Key Research Pointers ...Pointers ...

Instrument design architecture to ease Instrument design architecture to ease extraction of understanding architectureextraction of understanding architecture

Store architecture artifacts in schema-Store architecture artifacts in schema-based repository and as unstructured or based repository and as unstructured or Web-based text to ease searchingWeb-based text to ease searching

Allow for incomplete semantics and Allow for incomplete semantics and partial extraction of artifactspartial extraction of artifacts


Key Research Key Research Pointers ...Pointers ...

Allow user to build virtual, multiple Allow user to build virtual, multiple architectures, perspectives, and viewsarchitectures, perspectives, and views

Provide tools to compare virtual and Provide tools to compare virtual and code-centric architectures (e.g., code-centric architectures (e.g., reflection models [Murphy98])reflection models [Murphy98])

Make architecture extraction tools end-Make architecture extraction tools end-user programmable and extensibleuser programmable and extensible


Key Research Pointers ...Key Research Pointers ...

Develop methods and technology for Develop methods and technology for computer-aided data and database computer-aided data and database reverse engineeringreverse engineering• Integrate code and data reverse Integrate code and data reverse

engineering methods and toolsengineering methods and tools• Leverage synergy between code and data Leverage synergy between code and data

reverse engineering communitiesreverse engineering communities


Key Research Pointers ...Key Research Pointers ...

Develop tools that provide better Develop tools that provide better support for human reasoning in an support for human reasoning in an incremental and evolutionary reverse incremental and evolutionary reverse engineering process that can be engineering process that can be customized to different application customized to different application contextscontexts• End-user programmable toolsEnd-user programmable tools• Domain retargetable reverse engineeringDomain retargetable reverse engineering


Key Research Pointers …Key Research Pointers …

Concentrate on the tool adoption Concentrate on the tool adoption problem by improving the usability and problem by improving the usability and end-user programmability of reverse end-user programmability of reverse engineering tools to ease their engineering tools to ease their integration into actual development integration into actual development processesprocesses• Start with a web-based user interfaceStart with a web-based user interface• Conduct user studiesConduct user studies


ConclusionsConclusions

Mission statementMission statement• Researchers in software design and formal Researchers in software design and formal

methods should concentrate on software methods should concentrate on software evolution rather than constructionevolution rather than construction

• Program understanding and analysis experts Program understanding and analysis experts should teach their methods in 1st-yearshould teach their methods in 1st-year

Plenty of research problemsPlenty of research problems Wonderful case studiesWonderful case studies Exciting research!!!!Exciting research!!!!


Invitation to Visit Invitation to Visit CanadaCanada

May 12-19, 2001May 12-19, 2001

1 program comprehension and software migration strategies hausi a. müller university of victoria...

Documents

slices n design

patterns n business

technology models n

policies n function

source text slide

infrastructure slide

subject system

better engineered system