formal verification of storm topologies through...
TRANSCRIPT
DICEHorizon2020Research&InnovationActionGrantAgreementno.644869http://www.dice-h2020.eu
FundedbytheHorizon2020FrameworkProgrammeoftheEuropeanUnion
FormalVerificationofStormTopologiesthroughD-VerT
SA-TTAatSAC2017,MarrakechApr4th,2017
FrancescoMarconi,MarcelloM.BersaniandMatteoRossiDEIB, Politecnico di Milano, Italy
Roadmap
§ ContextandMotivation• StreamingApplication• ApacheStorm• DICEProject
§ D-VerT• Overview• Workflow• Usecase
§ Conclusion
2
CONTEXTANDMOTIVATIONFormalVerificationofStormTopologiesthroughD-VerT
3
Data-IntensiveApplications(DIAs)
o Needtoprocessdatabeing§ Massivelylargeinsize§ Complex§ Rapidlychanging
o DevotemostoftheirprocessingtimetoI/O,movementandmanipulationofdata.
o Relyonso-called"Bigdatatechnologies”
4
StreamingApplications
o SpecialcaseofDIAso Needtoprocessacontinuousflowofinformation
§ Streamà unboundedsequenceoftuples(messages)
o Usuallydescribedbymeansofatopology§ Graphofcomputationscomposedof
• input nodes(sourceofdatastreams)• computational nodesàmanipulatedatastreams
o Calculate,Filter,Aggregate,Join,Talktodatabases,etc
5
QualityIssuesinStreamingDIAso Importantrequirementsforstreamingapplications
§ Latency§ Throughput
o Criticalpoints§ incorrectdesignoftimingconstraints§ nodefailures
o mightcause§ Highlatencyinprocessingtuples§ Memorysaturation
6
latency
throughput
ApacheStorm
o OpenSourceDistributedStreamProcessingSystemo Analytics,LogEventprocessing,etc..o Reliability,at-least-onesemanticso Wideadoptioninproductiono InStormtopologies
§ Inputnodescalledspouts§ Computationalnodescalledbolts
7
DICEProject
o Horizon2020Research&InnovationAction(RIA)§ Quality-AwareDevelopmentforData-Intensiveapplications§ Feb2015- Jan2018,4MEurosbudget§ 9partners(Academia&SMEs),7EUcountries
8
Platform-IndependentModel
TheDICEApproach
9©DICE
Platform-SpecificModel
DICE profile
Deployment&ContinuousIntegration
DICE IDE
Big Data
ContinuousMonitoring
DeploymentModel
QAModels
ContinuousValidation
FormalVerification
D-VerT- DICEVerificationtool
o Assessmentofnon-functionalpropertiesof(streaming)DIAsatdesigntime
o Reliesonformalmodelingofstreamingapplications§ TemporalLogicModel(CounterNetworks)
o Aimsatcheckingthepresenceoftemporaldesignflaws§ Boltsnotabletoprocesstheincomingflowofmessages
10
D-VerTworkflow
11
- Create UML Model- Apply DICE Profile- Define Topology Structure- Apply Stereotypes- Provide values to parameters
MODEL
Run Verification
VERIFY
DICE-Profiled UML
Tool Configuration
Formal Model
TRANSFORM
✅
!REFINE
Property: “allbolts’inputbuffershaveaboundedoccupationlevel”
+
📈
Usecase:WebCrawler
o Goal:developawebcrawlingapplication§ Tofetch,parseandindexwebresources
• Input:continuousstreamofURLs• Output:indexedwebresources
§ Streamingparadigmwellsuitedforthistask
12
Usecase:WebCrawlerPlatform-IndependentModel(PIM)
13
o Goal:developawebcrawlingapplication§ Tofetch,parseandindexwebresources
• Input:continuousstreamofURLs• Output:indexedwebresources
§ Streamingparadigmwellsuitedforthistask
URLsStreamSource<<SourceNode>>
IndexingNode<<StorageNode>>
WebCrawlingApplication
<<ComputationNode>>
Usecase:WebCrawlerPlatform- SpecificModel(1)
14
Usecase:WebCrawlerPlatform-SpecificModel- Refinement
15
Usecase:WebCrawlerOutputtrace
16
CONCLUSIONS 17
Wrapup
oModel-drivenapproachtoformalverificationofstreamingapplications
o ApplicationdesignedandrefinedasannotatedUMLdiagrams§ SupportedbyD-VerTtool
• Basedontemporallogicmodel• automaticallygeneratesformalmodelsfromUMLmodelsandrunverification
• Checkwhetheranyofthebolts’queuesgrowsindefinitely
18
Futureworks
o Identificationandverificationoffurtherproperties
oModelingdifferenttechnologies§ Spark,CEP,…
o Toolandmodelimprovements
19
Questions?
20
Thankyou
21