talend metadata managerinfo.talend.com/rs/...en_di_talend_metadatamanager.pdf · environments have...

TalendMetadataManager

ReduceRiskandFrictioninyourInformationSupplyChain

TalendInc. 800BridgeParkway,Suite200,RedwoodCity,California94065USPage2Tel:+1(650)5393200

TalendMetadataManagerTalend Metadata Manager provides a comprehensive set of capabilities for all facets ofmetadata management. At the heart of Talend Metadata Manager is a repository whichcontains repository objects, such asmodels andmappings that are organized into folders.Models can be harvested from TalendData Integrationmodels, DataModeling tools, DataWarehouses, external metadata repositories for relational databases (RDBMS), and DataIntegration and Business Intelligence tools. A particular type of repository object calledConfiguration,canconnect“metadatastitching”modelsandmappingstogethertorepresentanEnterpriseArchitecture,includingfullsupportfordataflowlineageandimpactanalysis,aswellassemanticlineagedefinitions.

TalendMetadataManagerconsistsoffourmajorcomponents:

• MetadataBridge(metadataimport)• MetadataManager• DataGovernance• MetadataAuthoringwithForwardEngineering(metadataexport)


MetadataBridge

Metadataiseverywhere.Datawarehousing,businessintelligence,CASEandETLtoolsallhavetheirownrepositories.Justabouteveryapplicationhasitsowndatadictionary.XMLcarriesthe metadata with it in the message or document, and enterprise application integrationenvironmentshavetheirownrepositoriesandmetadatamappingandintegrationfacilities.Inordertosucceed,onemusthaveagoodenterpriserepositoryintegrationenvironmentthatcanintegratethedifferentformatofmetadatafromalltools.TheTalendMetadataManagerrepositorybridgesthetechnicalandnon-technicalaspectsofmetadata,whilesimultaneouslyaddressing the chasm between the different metadata source and target systems thatconstituteanymoderninformationmanagementenvironment.The Metadata Bridge imports all metadata via “bridges” (metadata import components),including Extract, Transformation and Load (ETL)/ Data Integration tools, BusinessIntelligencetools,DataModelingtools,databases,mostallmetadataexchangestandards,andnumerousdataformatsincludingXML.

ImportingmetadatafromTalendStudiowithTalendMetadataManager


MetadataManager(MM)

VersionandConfigurationManagementNotonlymusttherepositorybeabletoimportondemandinanyformatandtoanytoolorimportmetadatamanytimesasneeded,itmustbeabletomanagetheversionscreatedbythiscontinuous activity. It must also be fundamental to the repository organization foradministrators to then organize, publish and selectively present the information inappropriateconfigurationsofmetadata,asisrequiredforthecorrectandpreciseanswerstoawiderangeof“cuts”acrossthismetadata.TalendMetadataManagerwasdesignedfromthegroundupwithversionandconfigurationmanagementasakeycapability.

MetadataComparisonAllmetadataisrepresentedbyanintegratedmetamodelinTalendMetadataManager.Thisfeatureprovidescomparisonsacrossmetadatafromdatasourceformatssupported,includingdesigntools,databases,etc.,notsimplyamongversionsofagivenmodel.

ComparingmodelsormodelversionswithTalendMetadataManager


DataMappingSpecificationsOnceimported,metadatacanbemappedinamyriadofwaystoanyothermetadatawithinTalendMetadataManager.Thisabilityiscriticaltothesuccessofanymetadatamanagementsolution. Inparticular, youcandefinedata flowmappings describingdatamovement typerelationships,e.g.whenadatabaseisreadandtheresultswrittentoanotherdatabase,aswellas semanticmappingswhich identify semantic relationships between elements, oftentimesconceptualorlogicalinnature,suchasforadatadictionaryorconceptualmodelsuchasaUMLmodel.

MetadataStitchingMetadatastitchingisfundamentaltothecorrectandautomatedanalysisofthedataflowandsemanticlineageofmetadataintherepository.Italsosupportsversionmanagementacrosstheconstantrateofupdatesandchangesinarepository.TalendMetadataManagerkeepscompleteversionsofallimportedmetadatainself-contained“models”,whicharethenrelatedviastitching’s(simpleconnectionmappings). Inthisway,versionmanagementandconfigurationmanagement isnotonlyentirelycleanandisolatedfromthedefinitionandmaintenanceofmappings,italsoautomaticallysupportsupdatesandchangesintothefuture.

Gettingahighlevelviewofinformationflowsacrosssystemswithmetadatastitching


In this way, the enterprise architecture is correctly modeled, and data flow lineage iscompletelyandaccuratelyderivable.

Thedifferentrolesandtheirneedswithrespecttodataandrelatedmetadata

LineageandImpactAnalysisOncemetadata ismanaged,metadata is then available for detailed technical and businessanalysis. TalendMetadata Manager supports full technical and business level lineage andimpactanalysisprovidingyounewinsightacrossalltheconnectedmetadatasources.

BusinessUser–LineageReportinganalysisisthetypicalusecase,withquestionssuchas:

• Givenanitemonareport,whatdataentrysystemfieldsimpacttheseresults?• Whyarethenumbersonthisreportthewaytheyare?• HowdoIchangethesystemdatatocorrecttheresultsofthisreport?

DatalineagewithTalendMetadataManager


TechnicalUser–ImpactAnalysisOfhighinteresttothetechnicaluserarequestionslike:

• IfImustchangetheseelements(datatype,codesets,etc.)inmyoperationaldatastore,whatisthedownstreamimpact?

• ThisnewETLprocessispopulatingmystagingwarehouseinnewways,howdoesthisimpacttheOLAPmodelinmyreportingservices?

TechnicalUser–LineageReverselineagetypequestionsmayalsobeaskedbymoretechnicalusers,suchas:

• HowmanysystemsarerequiredtodeterminethedimensionsforthisportionoftheOLAPmodel?

• Abusinessreportusecase isaskingthe lineageforparticularvaluesonareport,sowheredoesthedatacomefromandhowisitmanipulated?

BusinessUsers–ImpactAnalysisFinally,businessusersmayasktheforwardlineageorimpactanalysisquestions,suchas:

• IfImakeachangetothisfield,whatreportswillbeimpacted?• How is this identity informationmergedwith the personnel system information on

theseotherreports?

ImpactanalysiswithTalendMetadataManager


DataGovernance(DG)

Critical to thedevelopmentandmanagementofa completedataarchitecture isaBusinessGlossary. Talend Metadata Manager provides an ISO 11179-based Business Glossary tocapture,define,maintainandimplementanenterpriseBusinessGlossaryofterminology,datadefinitions,codesets,domains,validationrules,etc.Inaddition,semanticmappingsdescribehowelementsinasourceModel(moreconceptualliketheBusinessGlossary)defineelementsinadestinationModel(closertoanimplementationorrepresentation).TheBusinessGlossaryhelpsanenterprisereachagreementbetweenallstakeholdersontheirbusiness assets (e.g. terms) and how they relate to data assets (e.g. database tables) andtechnology assets (e.g. ETL mappings). The Business Glossary can be used to documentlogical/physicaldataentitiesandattributesacrossITcollaboratively.Again,itinvolvestracingdependenciesbetweenbusinessandtechnicalassets.InTalendMetadataManager,aBusinessGlossaryisaself-containedcollectionofcategoriesand the terms sub-categories containedwithin each category. In turn, the termsmay besemantically mapped to objects throughout the rest of the repository, such as tables andcolumns inadatamodel. Oncemapped,onemayperformsemantic lineage tracessuchasdefinitionlookupsandtermsemanticusageacrossanyconfigurationscontainingtheBusinessGlossary,mappingsandmappedobjects.

AuthoringthecommonbusinesstermsusedintheorganizationwiththeBusinessGlossary


BootstrappingaBusinessGlossaryBuildingaBusinessGlossarycanbeassimpleasdragginginanexistingwell-documenteddatamodel,viaimportfromothersources(aCSVfileformat),orcanbepopulateddirectlyviatheuserinterfaceduringtheprocessofclassifyingobjectsinotherdatastoremodels.Ingeneral,acombinationofsuchmethodsareemployedinconjunctionwithoneanother.

WorkflowInordertoensurethattheBusinessGlossaryisaccurate,up-to-date,availabletoallwhoneedaccesstoit,andintegratedproperlywiththerestofthemetadataintherepository,TalendMetadata Manager also provides a robust collection of Data Governance tools andmethodologies. The Business Glossary provides a very flexible workflow and publicationprocessthatcanaddressbothbasicandcomplexneeds.Inaddition,onemaymaintainanynumberofbusinessglossaries,eachwithdifferentworkflowandpublicationcharacteristics.TheBusinessGlossarymaybepartofyourlineage.Itwillappearintherepositorypanelandwhen you open a Business Glossary, youwill be presentedwith a different UI than other(imported)Models.

Workflow-drivensearchcriteriaareavailableallowingonetoefficientlyorganizetermsandidentifywhatactionsarerequiredatanygiventime.Whenworkingwith individual terms,whichareatsomepoint intheworkflowprocess,workflowtransitionbuttonspromptyouwithpossibleactions.

SemanticMappingA SemanticMapping describes how elements in a sourcemodel (more conceptual) defineelementsinadestinationmodel(closertoanimplementationorrepresentation).Putanotherway, elements in the destination model are representations or implementations of theassociatedelementinthesourcemodel.Theyarethreeprimaryusesforsemanticmapping:

• DataStandardizationandCompliance• Multi Level Modeling of semantic relationships from conceptual to logical, and to

physicaldatamodelwithafewsubcases• BusinessGlossarytermclassification

TalendInc. 800BridgeParkway,Suite200,RedwoodCity,California94065USPage10Tel:+1(650)5393200WP208-EN

MetadataAuthoring(MA)withForwardEngineering(MetadataExport)Note:ThefollowingfeaturesonlycomewithTalendMetadataManagerwithAuthoring.

RDBMSandBigDataDocumenterandPhysicalDataModelerThe Talend Metadata Manager Data Documenter allows users to document existing datastores, like databases, big data sources, and imported models, and publish the resultingdocumenteddatastorestotheenterprise.TheDataDocumenteroffersadifferentapproachthantraditionaldatamodelingtools:

• The Business Glossary-driven Data Documentermethodology allows for immediatereuseandcreationoftermsandnamingstandardsonthefly,fasttrackingthedatastoredocumentationprocessensuringcompletesemanticsynchronizationamongyourdatamodelsanddatagovernanceenvironment.

• Web-enabledDataDocumenteroffersbetteraccesstousersthandesktoptools• DataModeling anddiagramming capabilities of theDataDocumenter are similar to

conventionaldatamodelingtools.• Fullintegration(import/export)tomostpopulardatamodelingtoolsisprovided.

VisualizingDataModelswithTalendMetadataManager


LogicalDataModelerTalend Metadata Manager provides a completely web-enabled logical data modelingenvironmentforproducinglogicalandconceptualmodels:

• TheBusinessGlossary-drivenmethodology allows for immediate reuse (creating ofentities,attributesanddomains)andcreationoftermsandnamingstandardsonthefly, fast tracking the modeling process and ensuring complete semanticsynchronizationamongyourmodelsanddatagovernanceenvironment.

• TheWeb-enabledmodeleroffersbetteraccesstousersthandesktoptools.• TheDataModelingcapabilitiesarecompetitivewithconventionaldatamodeling

tools.• Fullintegration(import/export)withmostpopulardatamodelingtoolsisprovided.

DataMappingDesignerData Mapping Designs represents data integration process designs containing all thenecessarydatamovementdesigndetails, such as lookups, filters, joins and transformationexpressions. TheseDataMappingDesignsare completeenough that theymaybe forwardengineered into Talend Data Integration using the Metadata Bridge. In this way, TalendMetadataManagerprovidesacompletelyweb-baseddatamappingdesigntoolthatcanreuseandbesynchronizedwithallothermetadataartifactsintherepositoryandyourcompletedatagovernanceenvironment.

DefiningthemappingsdirectlyinTalendMetadataManager


VisualizingtheendtoendinformationflowswithTalendMetadataManager

talend metadata managerinfo.talend.com/rs/...en_di_talend_metadatamanager.pdf · environments have...

Documents