connected corridors: i-210 pilot integrated corridor management … · 2019-12-17 · partners for...
TRANSCRIPT
![Page 1: Connected Corridors: I-210 Pilot Integrated Corridor Management … · 2019-12-17 · PARTNERS FOR ADVANCED TRANSPORTATION TECHNOLOGY INSTITUTE OF TRANSPORTATION STUDIES UNIVERSITY](https://reader034.vdocuments.site/reader034/viewer/2022042115/5e92e86bb87eda30616dae3b/html5/thumbnails/1.jpg)
PARTNERS FOR ADVANCED TRANSPORTATION TECHNOLOGY INSTITUTE OF TRANSPORTATION STUDIES UNIVERSITY OF CALIFORNIA, BERKELEY
Connected Corridors: I-210 Pilot Integrated Corridor Management System
Core System High-Level Design
November 29, 2017 DRAFT
Partners for Advanced Transportation Technology works with researchers, practitioners, and industry to implement transportation research and innovation, including products and services that improve the efficiency, safety, and security of the transportation system.
![Page 2: Connected Corridors: I-210 Pilot Integrated Corridor Management … · 2019-12-17 · PARTNERS FOR ADVANCED TRANSPORTATION TECHNOLOGY INSTITUTE OF TRANSPORTATION STUDIES UNIVERSITY](https://reader034.vdocuments.site/reader034/viewer/2022042115/5e92e86bb87eda30616dae3b/html5/thumbnails/2.jpg)
I-210Pilot:CoreSystemHigh-LevelDesign
ii
Thispageleftblankintentionally
![Page 3: Connected Corridors: I-210 Pilot Integrated Corridor Management … · 2019-12-17 · PARTNERS FOR ADVANCED TRANSPORTATION TECHNOLOGY INSTITUTE OF TRANSPORTATION STUDIES UNIVERSITY](https://reader034.vdocuments.site/reader034/viewer/2022042115/5e92e86bb87eda30616dae3b/html5/thumbnails/3.jpg)
I-210Pilot:CoreSystemHigh-LevelDesign
iii
![Page 4: Connected Corridors: I-210 Pilot Integrated Corridor Management … · 2019-12-17 · PARTNERS FOR ADVANCED TRANSPORTATION TECHNOLOGY INSTITUTE OF TRANSPORTATION STUDIES UNIVERSITY](https://reader034.vdocuments.site/reader034/viewer/2022042115/5e92e86bb87eda30616dae3b/html5/thumbnails/4.jpg)
I-210Pilot:CoreSystemHigh-LevelDesign
iv
TableofContents
1. Introduction ...................................................................................................................................1
1.1. PurposeofDocument ............................................................................................................. 1
1.2. RelationtoSystemsEngineeringProcess................................................................................ 1
1.3. IntendedAudience.................................................................................................................. 2
1.4. DocumentOrganization .......................................................................................................... 2
2. SystemPrimaryObjectivesandPurpose.........................................................................................4
2.1. ProjectGoalsandObjectives .................................................................................................. 5
2.2. TechnicalCapabilitiesSought.................................................................................................. 7
3. HighLevelDesignObjectives,Constraints,andPrinciples ...............................................................8
4. CoreSystemHighLevelDesign .....................................................................................................10
4.1. MajorComponents ............................................................................................................... 10
4.2. FieldElements ....................................................................................................................... 11
4.3. DataHub ............................................................................................................................... 12
4.4. DecisionSupport ................................................................................................................... 13
4.5. CorridorManagement .......................................................................................................... 14
4.6. Primaryprocessflow............................................................................................................. 17
5. DataHubDesign...........................................................................................................................18
5.1. DataSources ......................................................................................................................... 21
5.2. DataPipelines........................................................................................................................ 23
5.2.1. SensingDataPipeline .......................................................................................... 235.2.2. CaltransFreewayTrafficDataPipeline................................................................ 245.2.3. HeterogeneousDataPipeline.............................................................................. 255.2.4. HomogenousDataPipeline ................................................................................. 275.2.5. PipelineControl ................................................................................................... 275.2.6. PipelineStatusandLogging................................................................................. 295.2.7. CorridorManagementSystem-DecisionSupportSystem(CMS-DSS)Pipeline ... 305.2.8. CMSCommandStatusPipeline ........................................................................... 34
5.3. ExternalInterface/DataGateway.......................................................................................... 35
5.3.1. Subscriptionendpoints........................................................................................ 375.3.2. Request/Responseendpoints ............................................................................. 37
5.4. DataHubCommandGateway............................................................................................... 37
5.4.1. Conductor............................................................................................................ 38
![Page 5: Connected Corridors: I-210 Pilot Integrated Corridor Management … · 2019-12-17 · PARTNERS FOR ADVANCED TRANSPORTATION TECHNOLOGY INSTITUTE OF TRANSPORTATION STUDIES UNIVERSITY](https://reader034.vdocuments.site/reader034/viewer/2022042115/5e92e86bb87eda30616dae3b/html5/thumbnails/5.jpg)
I-210Pilot:CoreSystemHigh-LevelDesign
v
5.4.2. Camel................................................................................................................... 395.4.3. ActiveMQWorkflowStatusTopic ....................................................................... 395.4.4. ActiveMQWorkflowTaskTopic .......................................................................... 395.4.5. Monitor ............................................................................................................... 39
6. DecisionSupportSystemDesign...................................................................................................39
6.1. DSShighleveldesign............................................................................................................. 40
6.2. ResponsePlanManagement................................................................................................. 41
6.3. RulesEngine .......................................................................................................................... 43
6.4. Modeling ............................................................................................................................... 43
6.4.1. Modelingtechniques........................................................................................... 43
![Page 6: Connected Corridors: I-210 Pilot Integrated Corridor Management … · 2019-12-17 · PARTNERS FOR ADVANCED TRANSPORTATION TECHNOLOGY INSTITUTE OF TRANSPORTATION STUDIES UNIVERSITY](https://reader034.vdocuments.site/reader034/viewer/2022042115/5e92e86bb87eda30616dae3b/html5/thumbnails/6.jpg)
I-210Pilot:CoreSystemHigh-LevelDesign
1
1. INTRODUCTION
ThisConnectedCorridorsHighLevelDesigndocumentprovidesthehighlevelsystemarchitectureforthesystemtobedeployedontheI-210corridor.ThesystemarchitecturedescribedhereisadirectresultoftheConnectedCorridorsSystemRequirementsdocumentandtheworkdoneatUCBerkeleyintrafficmodelingandcontrol.Thisdocumentprovidesthesystemarchitecture,highleveldesignoftheprimarysubsystems,thedecisions,assumptions,constraints,andreasoningbehindthatarchitecture,andcriticalfunctionseachsubsystemprovidesforthesystem.
Thesystem,tobepilotedalongasectionoftheI-210corridorintheSanGabrielValleyareaofLosAngelesCounty,aimstoimproveoverallcorridorperformanceduringincidents,unscheduledevents,andplannedevents.Thisistobeachievedbymoreefficientlymanagingexistingsystemsandinfrastructures,promotingcross-jurisdictionaloperations,andusingmulti-modaltrafficanddemandmanagementstrategiesthatconsiderallrelevantmodesoftransportation.
1.1. PURPOSEOFDOCUMENT
Thisdocumentprovidesthehighleveldesign,servingastheidentificationoftheprimarysubsystemsandmajorcomponentsaswellasthebasisfortheselection,development,andintegrationoftheseintoasystemthatsatisfiesthesystemrequirementsasdefinedintheSystemsRequirementsDocument.ThishighleveldesignwillgovernthetechnologyplatformanddirectionoftheI-210PilotICMSystemandserveasthebasisforotherCaltrans-ledICMeffortsstatewide.
1.2. RELATIONTOSYSTEMSENGINEERINGPROCESS
ThedevelopmentofhighleveldesignispartofthesystemsengineeringprocessthattheFederalHighwayAdministration(FHWA)requiresbefollowedfordevelopingIntelligentTransportationSystem(ITS)projectswhenfederalfundsareinvolved.Whilenotrequiredforprojectsonlyusingstateorlocalfunds,useofthesystemsengineeringprocessisstillencouragedinsuchcases.
TheoverallsystemsengineeringprocessisillustratedinFigure1-1.DevelopinghighleveldesignrepresentsthenextstepoftheSystemDefinitionandDesignphaseofaproject(Phase2inthefigure)followingthecompletionoftheSystemRequirements.HighLevelDesignistypicallyderivedfromtherequirements.Theresultingdesignelementsareinturnusedtoinformandguidethemoredetaileddesignofthevarioussystemandsubsystemcomponents.
![Page 7: Connected Corridors: I-210 Pilot Integrated Corridor Management … · 2019-12-17 · PARTNERS FOR ADVANCED TRANSPORTATION TECHNOLOGY INSTITUTE OF TRANSPORTATION STUDIES UNIVERSITY](https://reader034.vdocuments.site/reader034/viewer/2022042115/5e92e86bb87eda30616dae3b/html5/thumbnails/7.jpg)
I-210Pilot:CoreSystemHigh-LevelDesign
2
Figure1-1–SystemRequirementsSpecificationwithinSystemsEngineeringProcess
1.3. INTENDEDAUDIENCE
TheprimaryaudiencefortheSystemHighLevelDesigndocumentincludespersonnelresponsiblefordesigningandimplementingtheICMsystem.TheaudiencealsoincludesindividualsfromCaltransDistrict7,CaltransHeadquarters,andtheUniversityofCalifornia,Berkeley,taskedwithprojectmanagementduties.
1.4. DOCUMENTORGANIZATION
Theremainderofthisdocumentisorganizedasfollows:
• Section2summarizestheprimarysystemobjectivesidentifiedwithintheSystemRequirementsthatshapethesystemdesign.
• Section3presentstheprimaryguidingdesignprinciplesandbaseassumptionsthatshapethesystemdesign.
• Section4presentsthekeysystemdesigncomponentsandprimarydataflows.
![Page 8: Connected Corridors: I-210 Pilot Integrated Corridor Management … · 2019-12-17 · PARTNERS FOR ADVANCED TRANSPORTATION TECHNOLOGY INSTITUTE OF TRANSPORTATION STUDIES UNIVERSITY](https://reader034.vdocuments.site/reader034/viewer/2022042115/5e92e86bb87eda30616dae3b/html5/thumbnails/8.jpg)
I-210Pilot:CoreSystemHigh-LevelDesign
3
• Section 5 presents the key system components of the Data Hub including data sources,interfaces/gatewaysandpipelines.
• Section6presentsthekeysystemcomponentsoftheDecisionSupportSystem(DSS)includingtherulesengine,modelinginterfaces,andresponseplangeneration.
• Section7presentskeysecuritydesignissuesandimplementationplans.
• Section8providesdesigninformationforthemessagingsystemsanddescribeshowinformationispassedbetweensubsystems.
• Section9describesthesysteminterfaces.
Inaddition,othersupportingdocumentsareavailableintheDocumentLibraryoftheI-210Pilotwebsiteathttp://ccdocs.berkeley.edu/content/document-library:
![Page 9: Connected Corridors: I-210 Pilot Integrated Corridor Management … · 2019-12-17 · PARTNERS FOR ADVANCED TRANSPORTATION TECHNOLOGY INSTITUTE OF TRANSPORTATION STUDIES UNIVERSITY](https://reader034.vdocuments.site/reader034/viewer/2022042115/5e92e86bb87eda30616dae3b/html5/thumbnails/9.jpg)
I-210Pilot:CoreSystemHigh-LevelDesign
4
2. SYSTEMPRIMARYOBJECTIVESANDPURPOSE
TheoverridingpurposeoftheI-210PilotistoreducecongestionandimprovemobilityalongasectionoftheI-210corridorinLosAngelesCountythroughthecoordinatedmanagementofitsmajornetworks:theI-210freeway,keysurroundingarterials,andlocalandregionaltransitservices.Thegoalistoenableallcorridor“actors”—transportationsystemmanagersandoperators,controlsystems,vehicles,andtravelers—toworktogetherinanefficientandcoordinatedway.
TheseimprovementswillbeachievedbydevelopinganddeployingtheICMsystemdescribedinthishighleveldesigndocument.AttheheartoftheproposedsystemwillbeaDecisionSupportSystem(DSS)designedtohelpcorridorsystemoperatorsmanageincidents,unscheduledevents,andplannedeventsmoreeffectively.Thissystemwilluseinformationgatheredfrommonitoringsystemsandprovidedbypredictiveanalyticaltoolstoestimatecurrentandnear-futureoperationalperformance.Theinformationwillbeusedtodeveloprecommendedcoursesofactiontoaddressproblemscausedbyidentifiedincidentsandevents.Morespecifically,thissystemisexpectedto:
• Improvereal-timemonitoringoftravelconditionswithinthecorridor• Enableoperatorstobettercharacterizetravelpatternswithinthecorridorandacrosssystems• Providepredictivetrafficandsystemperformancecapabilities• Beabletoevaluatealternativesystemmanagementstrategiesandrecommenddesiredcourses
ofactioninresponsetoplannedevents,unscheduledevents,andincidents• Improvedecision-makingbytransportationsystemmanagers• Improvecollaborationamongagenciesoperatingtransportationsystemsinthecorridor• Improvetheutilizationofexistinginfrastructureandsystems• Moreefficientlyusesparecapacitytoaddressnon-recurringcongestion• Reducedelaysandtraveltimesalongfreewaysandarterials• Improvetraveltimereliability• Helpreducethenumberofaccidentsoccurringalongthecorridor• Reducetheperiodduringwhichthecongestionresultingfromanincidentoreventaffects
corridoroperations• Reducegreenhousegasemissions• Generatehighertravelersatisfactionrates• IncreasetheoveralllivabilityofcommunitiesinandaroundtheI-210corridor
WhiledevelopmentoftheproposedsystemisunderthefinancialsponsorshipofCaltransHeadquarters,thesystemwillbedevelopedprimarilybythelocaltransportationagenciesthathaveagreedtoparticipateinitsoperation,incoordinationwithPATH.Projectactivitieswillincludethedesign,development,installation,testing,andoperationofvariouscomponentsoftheICMsystem,aswellasthedevelopmentofinterfaceswithexistingmonitoringandcontrolsystems.Forexample,theICMCoreSystemwillbeinterfacedwithtrafficmanagementsystemsownedbyCaltrans,suchastheAdvancedTrafficManagementSystem(ATMS).
![Page 10: Connected Corridors: I-210 Pilot Integrated Corridor Management … · 2019-12-17 · PARTNERS FOR ADVANCED TRANSPORTATION TECHNOLOGY INSTITUTE OF TRANSPORTATION STUDIES UNIVERSITY](https://reader034.vdocuments.site/reader034/viewer/2022042115/5e92e86bb87eda30616dae3b/html5/thumbnails/10.jpg)
I-210Pilot:CoreSystemHigh-LevelDesign
5
2.1. PROJECTGOALSANDOBJECTIVES
TheprimarygoaloftheI-210PilotICMprojectistoimproveoverallcorridorperformancealongasectionoftheI-210corridor.Thistranslatesintothefollowingspecificgoals:
1. Improveoperationalsituationalawareness2. Promotecollaborationamongcorridorstakeholders3. Improveresponsetoincidentsandevents4. Improvetravelreliability5. Improveoverallcorridormobility6. Empowertravelerstomakeinformedtraveldecisions7. Facilitatemulti-modalmovementsacrosstheregion8. Promotetransportationsustainabilitybyreducingimpactsontheenvironment9. Improvecorridorsafety
Foreachofthesegoals,Table2-1furtheridentifiesthemainoperationalobjectives.Manyoftheobjectivesaresimilartothoseoftraditionaltransportationimprovementprojects.Many,however,alsofocusonimplementingmorecomprehensivetravelandsystemstatusmonitoringsystems,improvedoperationalforecasting,improvedinformationdisseminationtotravelers,enhanceddata-sharingcapabilities,demandmanagementapproaches,andimprovedcollaborationamongtransportationsystemoperators.
Table2-1–ICMSystemGoalsandObjectives
Goals Objectives
1.Improvesituationalawareness
• Establishminimumrequirementsfordatacollectiontosupportsystemmanagement• Increasedatacollectionopportunitiesfromarterialsandlocalroads• Improvethecollectionofreal-timeoperationaldatafromnon-traditionalsources,suchasprobevehicles
• Developacomprehensivecorridorinformationaldatabasecoveringallrelevanttravelmodeswithinthecorridor
• Improvethequality,accuracy,andvalidationprocessofcollecteddata• Increasetheabilitytoestimatetraveldemandpatternsinamulti-modalenvironment• Improvetheabilitytoforecastnear-futuretravelconditionsbasedonknownincidents,roadconditions,weather,andlocalevents
• Developperformancemetricsconsideringallavailabletravelmodes
2.Promotecollaborationamongcorridorstakeholders
• Strengthenexistingcommunicationchannelsamongthecorridor’sinstitutionalstakeholders• Exploreopportunitiesfornewcommunicationlinksbetweencorridorstakeholders• Improvecooperationandcollaborationamongcorridorstakeholders• Developregional/jointoperationsconcepts• Identifynewmethodsofcollaboration• Extendcorridorperformancemetricstothenetworklevel• Investigatenewtypesofagreementsbetweenparticipatingagencies
![Page 11: Connected Corridors: I-210 Pilot Integrated Corridor Management … · 2019-12-17 · PARTNERS FOR ADVANCED TRANSPORTATION TECHNOLOGY INSTITUTE OF TRANSPORTATION STUDIES UNIVERSITY](https://reader034.vdocuments.site/reader034/viewer/2022042115/5e92e86bb87eda30616dae3b/html5/thumbnails/11.jpg)
I-210Pilot:CoreSystemHigh-LevelDesign
6
Goals Objectives
3.Improveresponsetoincidentsandunexpectedevents
• Reducethetimeneededtoidentifytheexistenceofanincidentorunexpectedevent• Reducethetimeneededtorespondtoincidentsorunscheduledevents• Enhancethecoordinationofactivitiesamongfirstresponders,trafficmanagementagencies,andtransitagenciestominimizeimpactsonsystemoperations
• Reducethetimeneededtoimplementcontrolactionstoaddresscongestionresultingfromanincidentorevent
• Reducethetimeneededtodisseminaterecommendeddetoursaroundanincidentorevent
4.Improvetravelreliability
• Improvetraveltimepredictabilityalongthecorridor• Reducetheimpactsofincidentsandeventsonnetworkoperations• Improveincident/eventnotificationforfirstrespondersandnetworkoperators• Improveincident/eventnotificationtotravelersandfleetoperators• Providetravelersandcommercialvehicleoperatorsaffectedbyanincidentoreventanenhancedabilitytoseekalternateroutesormodeoftransportation
5.Improveoverallcorridormobility
• Reducedelaysincurredbytravelers• Reducetheimpactsofincidentsandeventsonnetworkoperations• Efficientlyusesparecapacityalongcorridorroadwaystoplannecessarydetoursaroundincidentsorevents
• Promotestrategiestoinducedesirabletraveldemandpatterns• Coordinatethemanagementoffreewayandarterialbottlenecks• Promoteincreasesinvehicleoccupancy• Promoteincreasesintransitridership
6.Empowersystemuserstomakeinformedtraveldecisions
• Improvethedisseminationofreal-time,multi-modaltravelinformation• Enhancetheuseofinfrastructure-basedinformationaldevices(freewayCMS,arterialtrailblazersigns,kiosks,etc.)toprovideen-routeinformationtotravelers
• Enableindividualstoreceivetravelinformationonconnectedmobiledevices• Makearchivedhistoricaldataavailableto511servicesandinformationserviceproviders• Supportthedisseminationoftravelinformationby511servicesandthird-partyproviders
7.Facilitateregionalmulti-modalmovements
• Promotetheintegrationofcommuterrailandbusserviceswithcorridoroperations• Facilitatetransfersacrossmodesduringincidentsandevents• Providerelevantregionaltravelinformationtotravelers• Directtravelerstopark-and-ridefacilitieswithavailablespaces
8.Promotetransportationsustainability
• Reducefuelconsumption• Reducevehicleemissions• Identifyfinanciallysustainablesolutionsforlong-termsystemoperationsandmaintenance• Encouragetheuseoftransit,walking,andbicyclingwhereappropriate• Supportlocallypreferredalternativescompatiblewithcorridorobjectives• Developandimplementperformancemetricsreflectingenvironmentalgoals
9.Improvecorridorsafety
• Reducecollisionrates• Reducetheseverityofcollisions• Reducethenumberoffatalities• Reducetheimpactsofprimaryandsecondaryincidentsonnetworkoperationsthroughimprovedincidentmanagement
• Improvesafetyforbicycles,pedestrians,andtransit
![Page 12: Connected Corridors: I-210 Pilot Integrated Corridor Management … · 2019-12-17 · PARTNERS FOR ADVANCED TRANSPORTATION TECHNOLOGY INSTITUTE OF TRANSPORTATION STUDIES UNIVERSITY](https://reader034.vdocuments.site/reader034/viewer/2022042115/5e92e86bb87eda30616dae3b/html5/thumbnails/12.jpg)
I-210Pilot:CoreSystemHigh-LevelDesign
7
2.2. TECHNICALCAPABILITIESSOUGHT
Tohelpmanagetravelactivitieswithinthecorridorduringincidents,unscheduledevents,andplannedevents,theprojectisseekingthefollowingtechnicalcapabilitiestosupportthegoalsandobjectivesidentifiedinSection2.1:
• Gatherandarchiveinformationcharacterizingtrafficoperations,transitoperations,andtheoperationalstatusofrelevantcontroldeviceswithintheI-210corridor.
• IdentifyunusualtravelconditionsontheI-210freewayornearbyarterialsbasedonmonitoringdataprovidedbyvarioustraffic,transit,andtravelmonitoringsystems.
• Identifysituationsinwhichanincidentonroadwaysortransitfacilitiessignificantlyaffectstravelconditionswithinthecorridor.
• Providecorridor-wideoperationalevaluationstotrafficmanagers,transitdispatchers,andotherrelevantsystemmanagers,includingprojectedassessmentsofnear-futuresystemoperationsundercurrentandalternatecontrolscenarios.
• Identifyrecommendeddetoursaroundincidentsorroutesleadingtothesiteofanevent,consideringobservedtravelconditionswithinthecorridor.Dependingontheneed,andfinalsystemcapabilities,specificdetoursmayberecommendedformotoristsandtransitvehicles.
• Identifyrecommendedsignaltimingplanstouseatsignalizedintersectionstoimproveand/oraccommodatetrafficflowinfluxduringincidentsandeventsandimproveoverallcorridormobility.
• IdentifyrecommendedrampmeteringratestouseonindividualI-210freewayon-rampsandconnectorstomaintainoverallcorridormobility.
• IdentifymessagestopostonavailablefreewayandarterialCMSstoinformmotoristsofincidentsandevents.
• ProvideguidancetomotoristsontheI-210freewayandsurroundingarterialsusingavailablefreewayCMSs,arterialCMSs,andarterialdynamictrailblazersignsregardingwhichdetourtotaketogoaroundanincidentorwhichroutetofollowtoreachthesiteofanevent.
• Provideinformationtomotoristsabouttheavailabilityofparkingandtransitservicestohelptravelersmakealternatemode-choicedecisions.
• Provideuniformtrafficmanagementstrategiesacrossjurisdictionalboundariesduringincidentsandevents.
• Provideinformationtomotoriststhroughthird-partyoutlets,suchas511services,navigationapplicationproviders,etc.
![Page 13: Connected Corridors: I-210 Pilot Integrated Corridor Management … · 2019-12-17 · PARTNERS FOR ADVANCED TRANSPORTATION TECHNOLOGY INSTITUTE OF TRANSPORTATION STUDIES UNIVERSITY](https://reader034.vdocuments.site/reader034/viewer/2022042115/5e92e86bb87eda30616dae3b/html5/thumbnails/13.jpg)
I-210Pilot:CoreSystemHigh-LevelDesign
8
3. HIGHLEVELDESIGNOBJECTIVES,CONSTRAINTS,ANDPRINCIPLES
Thecoresystemdesignisgovernedbyafewkeyprimarydesignobjectivesdictatedbytheprojectandtherequirements:
• Realtimeoperation–Thesystemmustoperateinanearrealtimeenvironment.Thetimebetweenanincidentoccurringandaresponseplanimplementationisacriticaltimeperiod.Responsetimesshouldbedictatedprimarilybythetimetonotificationandconfirmationoftheincident,andthetimetofullyimplementadecision.Thetimerequiredbythesystemtoprocessinformationshouldbeminimizedsoastohavethemaximumpositiveimpactoncorridoroperations.
• Speedtodecision–Thereisasignificantamountofdataprocessingrequiredtomaintainbothoperatorawarenessandtoensurethemodelingwithinthedecisionsupportsystemisupdatedwiththelatestavailableinformation.Dataprocessinganddecisionprocessingtimeshouldbeminimized.
• Qualityofdecision–Thequalityoftheresponseplansisadirectresultofthetrafficestimationquality,trafficpredictionquality,dataprocessingtime,dataquality,andrulesusedtogenerateresponseplans.
• Abilitytomeasureoutcomes–Theremustbeahighlevelofconfidenceinthesystemoutcomes,andtheseoutcomesmustbecontinuouslymeasuredandmonitoredwithprocessesinplacetoimprovethesystemseffectivenessovertime.
• Incrementaldeployment–Thesystemmustbeabletobedeployedincrementally,addingnewcapabilitiesovertime.
• Systemflexibility–Thesystemmustbebuilttobeflexible,asitisintendedasapilotthatisadjustedandmodifiedasexperienceisgainedwithoperationsandgrowasnewcapabilitiesareadded.Inaddition,thisflexibilitywillbekeytofutureimplementationsinothercorridorswithdifferentintegrationneedsanddifferentrequirements.Itisalsoexpectedthattransportationitselfwillbechangingradicallyoverthelifetimeofthesystem,soflexibilityiskeytoitslongtermviability.
• Secureoperations–Asthesystemhasthecapabilitytorequestchangestotrafficcontrolsacrossalarge,complexurbancorridor,securityofthesystemiscriticaltoitssuccess.
Coresystemdesignislimitedbythefollowingconstraints:
• AbilitytobeoperatedandmaintainedbyCaltranswithoutsignificantlicensingcosts–Thesystemmustbedesignedwithopensourcecomponentsasmuchaspossible.Itisnotdesiredtohavesignificantlicensingcostsrequiredforsubsequentdeploymentsforfuturecorridorsoroverlongperiodsoftime.ItisintendedthatCaltranswillbecapableofdeploying,maintaining,andoperatingthisandfuturedeployments.
• Limitedtimetodeploy–Thedeploymentscheduleisaggressive,creatingtheneedtoensuresignificantreusabilitywithinthedesignsothatmultipledatasourcescanbeincorporatedintothesystemwithoutdesigningnewdatapipelinesforeachone.
![Page 14: Connected Corridors: I-210 Pilot Integrated Corridor Management … · 2019-12-17 · PARTNERS FOR ADVANCED TRANSPORTATION TECHNOLOGY INSTITUTE OF TRANSPORTATION STUDIES UNIVERSITY](https://reader034.vdocuments.site/reader034/viewer/2022042115/5e92e86bb87eda30616dae3b/html5/thumbnails/14.jpg)
I-210Pilot:CoreSystemHigh-LevelDesign
9
• Constraineddevelopmentresources–Caltransprovidedafixedamountoffundingsothesystemdesignmustnotexceedourresourceswithcomplexityandadditionalfeatures.
Thesecoredesignobjectivesandconstraintsareimplementedwiththefollowingdesignprinciples:
• Clouddevelopment,deployment,andoperations–Inordertoachievemaximumflexibilityinbothsystemdevelopmentandfutureconfigurations,minimizeresourceutilizationindevelopmentanddeployment,andminimizedeploymenttime,thecoresystemisdesignedfor100%cloudoperationswithinanAmazonAWScloudenvironment.
• Thesystemisapilotsystemanditexpectedthataswelearnfromthedeploymentofthesystemchangeswillbeneededandrecommended.
• ThesystemshallbecapableofbeingsupportedbyCaltransinternalresources.• Decisionsproducedintheformofresponseplansshallbebasedoncurrentcorridorinformation
withlimiteddelaysininformationprocessing.• Datamaintainedwithinthecoresystemwillbelimitedtothedatarequiredforsystem
operation.LongtermdatastorageshallbeafunctionofPeMS.• Thesystemwillbedesignedforfuturegrowth,incrementalimprovements,future
transportationsystemchanges,changesintransportationitself,geographicchanges,andnewandupdatedinformationsourcesandneeds.
• Easeofdeployment• Easeofmaintenance• Portabilityofthesystemdesignandcomponentstoothercorridors• Highlydecoupleddesignforflexibility,scalability,redundancy,andreliability• Useavailabledatastandardswheneverpossible• Standardizedexternalinterfacesforeaseofportabilityandinclusionofnewsubsystems• Maintenanceofaseparationofconcernsbetweendatamanagement,decisionsupport,and
controlfunctions.• Designwithstate,regional,andlocallayersforfuturescalabilityandstandardizeddeployment
acrossthestate.
![Page 15: Connected Corridors: I-210 Pilot Integrated Corridor Management … · 2019-12-17 · PARTNERS FOR ADVANCED TRANSPORTATION TECHNOLOGY INSTITUTE OF TRANSPORTATION STUDIES UNIVERSITY](https://reader034.vdocuments.site/reader034/viewer/2022042115/5e92e86bb87eda30616dae3b/html5/thumbnails/15.jpg)
I-210Pilot:CoreSystemHigh-LevelDesign
10
4. CORESYSTEMHIGHLEVELDESIGN
ThecoresystemhighleveldesignisillustratedinFigure4-1.Thishighleveldesignprovidesclearseparationbetweenexternalsystemsprovidingdataandreceivingcontrolrequests(green),thedatahubreceivingandprocessinginformationfromthoseexternalsystems(red),decisionsupportprovidingresponseplansforincidents(blue),andthecorridormanagementsystemprovidinguserinterface,responseplanselectionandapproval,andsendingofcontrolrequeststoexternalsystems(purple).
Figure4-1CoreSystemHighLevelDesign
4.1. MAJORCOMPONENTS
AsshowninFigure4-1,theICMCoreSystemiscomposedofthreemajorsubsystems:theDataHub,theDecisionSupportSystem,andtheCorridorManagementsubsystem.Thesesubsystems,plustheexternalfieldelementstheCoreSysteminteractswith,aresummarizedinthefollowingtable(color-codedtomatchFigure4-1)anddescribedindetailinthefollowingpages.
Component Description
DataHub Providesforreceiptandprocessingofallcorridordataandamethodofcommunicationamongthethreesubsystems,usingacommonsetofdatadefinitions.Providesoperationaldatapersistenceandretrieval.
DecisionSupport Providescurrenttrafficstate,responseplandevelopment,andresponseplanperformancepredictiontoprovideoneormorerecommendedresponseplansforincidentsandevents.
CorridorManagement Providestheprimaryuserinterfaceforsystemoperators,amethodforusersto
![Page 16: Connected Corridors: I-210 Pilot Integrated Corridor Management … · 2019-12-17 · PARTNERS FOR ADVANCED TRANSPORTATION TECHNOLOGY INSTITUTE OF TRANSPORTATION STUDIES UNIVERSITY](https://reader034.vdocuments.site/reader034/viewer/2022042115/5e92e86bb87eda30616dae3b/html5/thumbnails/16.jpg)
I-210Pilot:CoreSystemHigh-LevelDesign
11
monitorcurrentcorridorandcorridorassetstates,interactwiththeDecisionSupportSystem,review,evaluate,select,andapproveresponseplans,andinteractwithexternalsystems,particularlyforexecutionandmanagementofresponseplans.
FieldElements Representexternalprovidersandconsumersofinformationandrequestsforactionsuchasintersectionsignals,rampmeters,corridorsensing,andotherelements,usuallythroughrespectivetransportationmanagementsystemsandTMCs
ThedesignfortheI-210Pilotwillnothavealayeredgeographicapproach,butthedesignisintendedtoallowsuchanapproachforfutureuse.Therearethreegeographiclayerstothedesign:state,regional,andlocal.TheintentisthattheDataHuboperatesonaregionallevel(Caltransdistrict),withthepotentialforservingmultiplecorridorswithinaregion.DatawouldbeconsolidatedandaggregatedatastatelevelformultipleregionsviaPeMS,tosupportlargerscalearchiving,businessintelligenceanalysis,andcontinuedimprovementofcorridorandICMsystemcapabilitiesatthestatelevel.TheDecisionSupportandCorridorManagementcomponentsoperateonalocallevel,withinasinglecorridor.Withstateconsolidationofdata,regionaldatacommunication,andlocaldecisionsupportandresponseplanexecution,amethodofsharinginformationbetweencorridorsisenabled,whileensuringlocaljurisdictioncontroloftrafficeventandincidentresponse.
4.2. FIELDELEMENTS
GreenelementsinFigure4-1representthevariouscorridorfielddatasourcesandfieldelementcontrols,includingvariousstate,regional,andlocaltransportationsystems,regionaltransportationdatanetworks,andprivateinformationproviders.
• Ontheleftsideofthediagram,theseelementsrepresentthevariousdatasourcesforinformationconsumedandprocessedbytheICMsystem.
• Ontherightside,theserepresentthevariouscontrolinterfacestoexecuteresponseplanelementsselectedbyoperatorsoftheICMsystem.
ThesystemsthatwillbeprovidingthedatawithintheI-210corridorinclude:
Table4-1–FIELDSYSTEMS
Source InformationTypePasadenaTMC* PasadenaintersectionsignalsArcadiaTMC* Arcadiaintersectionsignals
LACountyintersectionsignalsDuarteintersectionsignals
CountyTMC*
MonroviaintersectionsignalsFreewayloopsensing(traffic)Rampmeters
CaltransATMS**
Dynamicmessagesystem
![Page 17: Connected Corridors: I-210 Pilot Integrated Corridor Management … · 2019-12-17 · PARTNERS FOR ADVANCED TRANSPORTATION TECHNOLOGY INSTITUTE OF TRANSPORTATION STUDIES UNIVERSITY](https://reader034.vdocuments.site/reader034/viewer/2022042115/5e92e86bb87eda30616dae3b/html5/thumbnails/17.jpg)
I-210Pilot:CoreSystemHigh-LevelDesign
12
IncidentinformationCaltransLCS* LaneclosuresCaltransTSMSS* IntersectionsignalTrailblazerSystem* Dynamicmessagesystemforrerouting
LocalvideoRIITSCaltransvideoEnvironmentalsensingCalPolyODSTransit
HSRLaneClosure* CitylaneclosuresNextBus Goldlinetransit
*Indicatesfieldsystemthatacceptscontrolmessages
**Caltrans’ATMSsystemacceptscontrolmessagesandprovideslimitedICMcoresystemcontrol
Futurefieldsystemsmayincludeprobedataproviders(GPSlocation/speed),additionaltransitinformation,parkinginformation,andothers.
`
4.3. DATAHUB
DatafromeachofthefieldsourcesarereceivedbytheICMDataHub,representedinredinFigure4-1.TheprimaryfunctionsoftheDataHubare:
• Receivedatafromfieldelementsviaexistingcorridortrafficmanagementcentersandregionaldatanetworks:VariousdatareceiversreceivedataandprepareitforprocessingbytheICMsystem.Thesereceiversaregenerallyexpectedtobebuiltforthespecificinterfacesdefinedbyeachfielddatasource,bothintransmissionmethod(RESTorSOAPwebservices,socket-basedstreaming,file-based,orothers)andtheintendedinitialpathrequiredforsystemdataprocessing.ThepreferredmethodofinformationtransferforthesystemistheTrafficManagementDataDictionary(TMDD)standarddevelopedbytheInstituteforTrafficEngineers(ITE),currentlyatversion3.03d(withcertainmodifications).
• Processdatareceivedfromfieldelements:Datafromfieldelementsmustbevalidatedforcompletenessanddataqualitypriortousebydownstreamsystemcomponents.Withsuchavarietyofdatasources,oftenforthesametypeoffieldelements,datamustbetransformedintoacommonsetofdatasemantics.TheDataHubprocessesallincomingdataintoastandardizedformat,TMDDfortransportationassets,GTFSfortransitinformation,orothersdependinguponthedatabeingreceived,withacommonsetofdatadefinitionsaswell(suchasasinglenamingstandardforallstreetswithinthecorridor).However,itiscriticaltonotethatthistransformationintoastandardizedformatforprocessingwithinthedatahub,whileitmaintainsthedatastructureswithinTMDD,doesnotgenerallymaintainanXMLdataformatandinternalcommunicationswithinthedatahub(andtheDSS)donotuseSOAPprotocols.ThedatahubandDSSuseJMSprotocols,eitherwithinanActiveMQorKafkamessagingsystem.
![Page 18: Connected Corridors: I-210 Pilot Integrated Corridor Management … · 2019-12-17 · PARTNERS FOR ADVANCED TRANSPORTATION TECHNOLOGY INSTITUTE OF TRANSPORTATION STUDIES UNIVERSITY](https://reader034.vdocuments.site/reader034/viewer/2022042115/5e92e86bb87eda30616dae3b/html5/thumbnails/18.jpg)
I-210Pilot:CoreSystemHigh-LevelDesign
13
Additionalprocessingisalsocompletedwithinthedatahub,eitherforspecificmetrics,calculatedparameters,orpredictiveanalytics.
• Datamessagingandcommunications:Dataprovidedtodownstreamsystems,aswellasinternalsystemcommand,control,andstatusdata—indeed,alldatawithintheICMsystem—ismadeavailableviaaninternaldatabus.Themethodofdatatransportisviadatamessagingtechnologies,namelyActiveMQJMSmessagingorKafkadatamessagingsystems.Thespecificmessagetechnologyisbasedonthetype,size,frequency,andmessagepersistencerequirementsofthedata,dataproducer,anddataconsumers.Exceptionstothesemethodswithinthedatahubarelimited,andaregenerallyusedforlargeblockbulkdataarchiving/datatransportbetweenpersistencestores.InordertoaccommodatemultipleCMSvendors,andtoprovidecommunicationbetweenthedatahub’sKafkamessagingsystemandtheDSSActiveMQmessagingsystem,anApacheCamelbasedinterfacebetweenthedatahubandtheDSSandCMSsystemsisincludedwithinthedatahub.ThisinterfaceprovidestheabilitytoprovideSOAPandRESTbasedinterfacesasnecessary,andtranslationbetweenmessagingsystems.
• Datapersistence:TheDataHubalsoprovidesdatapersistencecapabilities,allowingforpersistenceofrawandprocesseddata,systemcommand/control/status,andothersysteminformationinacentralrepository.Thisdatapersistenceisbrokenintodifferenttimelayers,includinglivereal-timedata,youngdata(0-90days),aggregateddata,andarchiveddata.ItincludesleveragingstateEDWandBIresourcesasacomponentofthedatapersistencecapabilities,specificallyasthesingledatastorefornon-operationaldata(dataolderthan90days).Sincedatapersistencewithinthedatahubislimitedtooperationaluses,andthearchitectureisbasedonapatternofcoreservicesconnectedbymessaging,multipledatapersistencetechnologiesareutilizedspecifictotheneedsofthesystemandthedatabeingstored.Largetimeseriescollectionsofdata,suchassensordata,ismaintainedinaCassandradatabase;largerelationalstructureswithsmallerupdatefrequenciesaremaintainedwithinaPostgresdatabase,andMongoDBisusedwithindatatranslationpipelineswhenmultiplesourcesprovidethesametypeofinformationindifferingformatsandimplementations.NOTE:MongoDBmaybeusedindevelopmentandearlyversionsofthesysteminsteadofPostgresforsomeoftherelationaldatastores.
4.4. DECISIONSUPPORT
TheDecisionSupportSystem,portrayedinblueinFigure4-1,providesthefollowingcapabilities:
• Corridortrafficstatedetermination:TheDecisionSupportSystemprovidescorridortrafficstateestimation,providingbothgeospatialtrafficstateinformationandtrafficstatemetrics.Separatearterialandfreewaytrafficmodelsareusedandmergedtoprovidefullcorridortrafficstateatalltimes.
• Corridortrafficstateprediction:TheDecisionSupportSystemprovidescorridortrafficstatepredictionsforusebycorridoroperatorsandforpredictionofincidentandeventresponseplanperformance.Acommercialsimulationengine(TSSAimsun)isusedtoprovidetraffic
![Page 19: Connected Corridors: I-210 Pilot Integrated Corridor Management … · 2019-12-17 · PARTNERS FOR ADVANCED TRANSPORTATION TECHNOLOGY INSTITUTE OF TRANSPORTATION STUDIES UNIVERSITY](https://reader034.vdocuments.site/reader034/viewer/2022042115/5e92e86bb87eda30616dae3b/html5/thumbnails/19.jpg)
I-210Pilot:CoreSystemHigh-LevelDesign
14
predictions.Trafficpredictionsaregeneratedusingthetrafficestimatedstateasaninitialtrafficstate,andcurrentcorridorassetstatefromthedatahub.
• Responseplandevelopmentandevaluation:TheDecisionSupportSystem,uponnotificationofaconfirmedincident,willusearules-basedapproachalongwithtrafficstateestimationandpredictioncapabilitiestodevelop,evaluate,andrankresponseplansforusebycorridoroperators.
4.5. CORRIDORMANAGEMENT
TheCorridorManagementsubsystem,portrayedinpurpleinFigure4-1,shallprovidethefollowingprimarycapabilities:
• Presentationofcorridorstate:Displayofcorridorstate,includingcorridorassets(signals,ramps,cameras,dynamicmessaging,transit,etc.),trafficestimatedstate,trafficvisualization(cameraoutput),humanassets,physicalassets(vehicles,responsecrews).Assetstateincludescurrentoperationalcapabilities,currentoperatingstate,currentfunctionalstate(operating,failed,degradedstates),timeavailability,controllability,etc.
• Controlofcorridorassets:TheCorridorManagementsubsystemprovidesthecapabilitytocontrolcorridorassets,takingresponseplansapprovedforexecutionandexecutingeachoftheindividualresponseplanelementsbysendingcommandstotheappropriatestate,regional,andlocalsystemsandentities.TheCorridorManagementsubsystemdoesnotdirectlycontrolcorridorassets,butonlysendscommandstothestate,regional,orlocalsystemsthatdirectlycommandcorridorassets.
• Presentationofresponseplans:TheCorridorManagementsubsystempresentseachresponseplandevelopedbytheDecisionSupportSystem,displayinginmeaningfulwaysthevariousresponseplanelements,howtheywillchangefromthecurrentcorridorstate,howtheychangeduringresponseplanexecution,whatthepredictedoutcomesareforeachresponseplan,whattheactualoutcomesareforaresponseplan,aswellasthevariousmetricsonhowtheresponseplansareevaluatedandtheireffectivenessmeasured.Thispresentationisexpectedtobeinmultipledomains,includinggeospatial,tabular,comparative,graphical,andothersolution-specificmethods.
• Responseplanlifecyclemanagement:Eachresponseplandevelopedbythesystemhasaspecificlifecyclethatincludes:
1. Initiation2. Development3. Evaluation4. Selection5. Approval
6. Execution7. Monitoring8. Close9. Post-incident/eventanalysis
![Page 20: Connected Corridors: I-210 Pilot Integrated Corridor Management … · 2019-12-17 · PARTNERS FOR ADVANCED TRANSPORTATION TECHNOLOGY INSTITUTE OF TRANSPORTATION STUDIES UNIVERSITY](https://reader034.vdocuments.site/reader034/viewer/2022042115/5e92e86bb87eda30616dae3b/html5/thumbnails/20.jpg)
I-210Pilot:CoreSystemHigh-LevelDesign
15
TheCorridorManagementsubsystemprovidesamanagementinterfaceforthislifecycle:
Initiation TheCorridorManagementsubsystempresentsincidentandeventinformationreceivedfromtheDataHubforreview,edit,andconfirmationbythecorridoroperators.
Development TheCorridorManagementsystemprovidesdisplayofstatusinformationreceivedfromtheDecisionSupportSystem(DSS)regardingitsdecisiontodevelopasetofresponseplanstoanincidentorevent,aswellasthedevelopmentofthoseresponseplans.Inaddition,itcansendcommandstotheDecisionSupportSystemtoinitiatespecificDSSfunctions,suchasmockincidentevaluation,manualresponseplandevelopmentorevaluation,orotherfunctionsrequiredfortheICMsystem.
Evaluation TheCorridorManagementsubsystemreceivestheresultsofresponseplanevaluationandtrafficstatepredictionanddisplaysthoseresults,alongwiththeresponseplans,tothecorridoruser.Displayshallincludeindividualplananalysis,aswellasrankingandcomparativeanalysisofresponseplans.
Selection TheCorridorManagementsubsystemprovidestheuserthecapabilitytoselectaspecificresponseplanforsubsequentapprovalandimplementation.
Modification TheCorridorManagementsubsystemshallprovideuserstheabilitytomakeminormodificationstoresponseplans.NOTE:Initialversionsofthesoftwarewillnothavethiscapability.
Approval TheCorridorManagementsubsystemprovidesaselectedresponseplantothecorridorstakeholdersatthestate,regional,andlocallevelsforreviewandapproval.Theseapprovalsshallincludeminormodificationstotheresponseplanandmayincludereevaluationandresubmittalforapproval.
Execution TheCorridorManagementsubsystemisthesolecomponentinthesystemcapableofsendingcommandstothevarioussystemsincontrolofcorridorassetsforindividualresponseplancomponentexecution.Itshallhavedisplaysthatallowcontrolandmonitoringofcorridorassetsviathestate,regional,andlocaltrafficmanagementsystemsandassets.
Monitoring Giventhecapabilitytodisplayassetstate,managecorridorassets,andexecuteresponseplanelements,theCorridorManagementsubsystemshallprovideanintegrateddisplayofresponseplanmonitoringoncetheexecutioncommandsaresenttothefieldsystems.Monitoringshallincludedisplayingwhencommandshavebeenexecutedandarecompleteatthefieldassetlevel,alertinguserswhencommandsorassetshavefailed,identifyinganddisplayingvariancesbetweenexpectedandactualtrafficstate,andtheabilitytotakeactionintheeventofresponseplanelementfailureforanyspecificcorridorasset.
Close TheCorridorManagementsubsystemshallbeabletoreturncorridorassetstotheirnormalstateoncetraffichasreturnedtoanormalstate.
Post-incident/eventanalysis
TheCorridorManagementsubsystemshalldisplayacorridorpost-eventanalysisforeachincidentorevent.TheanalysisshallincludeinformationfromtheCorridorManagementsubsystemitself,especiallywithregardtothecorridorassetresponseplanexecution,responseplanexecutionissuesand
![Page 21: Connected Corridors: I-210 Pilot Integrated Corridor Management … · 2019-12-17 · PARTNERS FOR ADVANCED TRANSPORTATION TECHNOLOGY INSTITUTE OF TRANSPORTATION STUDIES UNIVERSITY](https://reader034.vdocuments.site/reader034/viewer/2022042115/5e92e86bb87eda30616dae3b/html5/thumbnails/21.jpg)
I-210Pilot:CoreSystemHigh-LevelDesign
16
failures,trafficanalysis(expectedvs.actual),andothersystemperformancemeasures.Thepost-eventanalysisisprimarilylimitedtoinformationusedbytheICMsystemanditscomponents,aswellasmetricscalculatedpostincident/eventregardingresponseplanandsystemperformance.Itisnotintendedtobeanexhaustiveanalysiswithextensiveadditionalmodelinganalysisand“what-if”scenarios.
• ICMSystemmanagement:TheCorridorManagementsubsystemprovidesmanagementcapabilitiesfortheICMsystem,including:
Rulesmanagement TheCorridorManagementsubsystemprovidesauserinterfaceforthemanagementoftherulesusedwithintherulesengine.Thiswillincludetheabilitytocreatebasicrulesandrulesets,editrulesandrulesets,archiverulesandrulesets,provideconfigurationmanagementforrulesandrulesets,makerulesactive,executerulesandrulesets,andimportrulesandrulesets.Generally,rulesthatcanbeeditedwillbesimplerrulesandrulessets,asmorecomplexrulesarelikelytorequireprofessionaldevelopmenteffortandintegration.NOTE:Theinitialimplementationofthesystemwillnotincludethiscapability.
Responseplanmanagement
TheCorridorManagementsubsystemprovidesuserstheabilitytoupdatetheavailableresponseplansandresponseplanelements.Thisincludesadding,archiving,updating,andactivationordeactivationofresponseplansorresponseplanelementsforusewithinthecorridor.
Systemmonitoring TheCorridorManagementsubsystemprovidesuserstheabilitytomonitortheICMsystemandcomponents.Systemslogs,alerts,statusinformation,controlandexecutionofsystemfunctionsshallbeaccomplishedbytheCorridorManagementsubsystem.
Security TheCorridorManagementsubsystemprovidesuserauthenticationandauthorizationfortheCorridorManagementsubsystemandallfunctionsavailablewithintheCorridorManagementsubsystemthataffectotherICMsystemfunctions,includingDecisionSupportandtheDataHub.ThisdoesnotincludeuserauthenticationandauthorizationofindividualcomponentssuchasCassandra,Postgres,ormessaging;rather,itprovidestheseservicesforthefunctionsthatusethosecomponents.
Configuration TheCorridorManagementsubsystemprovidesmethodsandinterfaceforallowinguserstochangeanyandallconfigurationelementswithintheICMsystem.Aswithsecurity,thisdoesnotincludeconfigurationofindividualsystemcomponentssuchasCassandraandPostgres,butratherforthefunctionsthatusethesecomponents.NOTE:Theinitialimplementationofthesystemwillhaveminimalcapabilitiesforthisfunction.
![Page 22: Connected Corridors: I-210 Pilot Integrated Corridor Management … · 2019-12-17 · PARTNERS FOR ADVANCED TRANSPORTATION TECHNOLOGY INSTITUTE OF TRANSPORTATION STUDIES UNIVERSITY](https://reader034.vdocuments.site/reader034/viewer/2022042115/5e92e86bb87eda30616dae3b/html5/thumbnails/22.jpg)
I-210Pilot:CoreSystemHigh-LevelDesign
17
4.6. PRIMARYPROCESSFLOW
Figure4-2providestheprimarysystemworkflowillustratingthebasicintegrationbetweeneachofthesubsystems.Whilethisillustrationisnotintendedtocaptureallofthedetailsoftheinteractionsbetweensystems,andtheprocesslifetimesarenotintendedtobeaccuratewithinthissequencediagram,itdoesprovidethebasicsequenceofeventsandprimarycommunicationchannelsfortheprimaryworkflowwithinthesystem,aresponsetoanincidentonthecorridor.Secondaryactionssuchaspersistence,logging,communicationdialogs,andothersarenotdepictedinthisdiagram.Alsonotethat,asdescribedinFigure4-1,allcommunicationbetweentheDSSandCMSisviatheDataHub’sdatabus,althoughthespecificsofthecommunicationchannelarenotillustratedinFigure4-2.
Figure4-2PrimarySystemIncidentFlow(Subsystem)
Thebasicworkflowdescribedinthisdiagramisasfollows:
• Anincidentoccurswithinthecorridor.Thediagramshowstheincidentinformationbeingprocessedthroughthedatahub.
• TheincidentisconfirmedbyanoperatorwithintheCMSorCaltran’sATMS.Theinitialpilotsystemwillalwaysuseanoperatortoconfirmanincident.
![Page 23: Connected Corridors: I-210 Pilot Integrated Corridor Management … · 2019-12-17 · PARTNERS FOR ADVANCED TRANSPORTATION TECHNOLOGY INSTITUTE OF TRANSPORTATION STUDIES UNIVERSITY](https://reader034.vdocuments.site/reader034/viewer/2022042115/5e92e86bb87eda30616dae3b/html5/thumbnails/23.jpg)
I-210Pilot:CoreSystemHigh-LevelDesign
18
• TheincidentinformationispassedtotheDSS’sresponseplandevelopmentcomponent.• Theresponseplandevelopmentcomponentrequestsapredictionoftheimpactoftheincident
onthecorridor.• Thepredictionengineusesthecurrentestimatedtrafficstateandcorridortrafficstateto
initializeandrunapredictionwithnoresponseplanexecution(“donothing”prediction).• Theresponseplandevelopmentcomponentusesthe“donothing”predictionandtherules
enginetodetermineifaresponseplanshouldbedeveloped.Iftheresultisnoresponseplanshouldbedeveloped,executionoftheworkflowwouldendwithnotificationtotheCMSofthatresult.
• Iftheresultisthataresponseplanshouldbedeveloped,theresponseplandevelopmentcomponentagainusestherulesengine,resultsofthe“do-nothing”prediction,currentcorridorstate,andasetoffixedresponseplancomponentstodevelopalimitednumberofresponseplansforevaluation.Itsubmitsthoseresponseplanstothepredictionengine.
• Thepredictionenginerunsapredictionforeachresponseplan,alongwithanother“donothing”prediction”basedoncurrentcorridorconditions.Itprovidestheresultsofthosepredictionstotheresponseplandevelopmentcomponent.
• Theresponseplandevelopmentcomponentusestheresultsofeachprediction,currentcorridorstate,andtherulesenginetoevaluate,rank,andselectarecommendedresponseplan.
• TheresultsofresponseplanpredictionsandevaluationisprovidedtotheCMSforoperatorselection.
• TheCMSprovidestheresultstoanoperator,whoselectstheresponseplantobeimplementedandobtainsapprovalfortheresponseplanimplementation.
• Iftheresponseplanisapproved,theCMSexecutestheresponseplanbysubmittingC2CcommandstotheTMCsandothersystemsrequiredtoexecutecommandstoindividualcorridorassets.
• TheindividualTMCsandothersystemsexecutingtheresponseplancommandsreportthesuccessorfailuretoexecutethedesiredresponseplanelements,andcontinuetoreportcorridorassetstate.
Uponcompletionofthisprimaryworkflow,furtherprocessesinvolvedinresponseplanlifecyclemanagementwilloccur,suchasresponseplanevaluation,responseplangenerationtoadjusttocurrentconditions,responseplancancellation,andresponseplancloseout.
5. DATAHUBDESIGN
Thedatahubhasfiveprimaryroleswithinthesystem:
1. Receiveinformationfromexternalprovidersofcorridorstate2. Processinformationreceivedfromprovidersofcorridorstate,evaluatingthedataforquality,
providingcommonmetricsandanalysisofthedatareceived,conductingpredictiveanalyticstosupportestimationandpredictionwithintheDSS,andstandardizingtheformatandcontentfordownstreamconsumptionbytheDSSandCMSsystems.
3. Persistinformationreceived,resultsofprocessing,andoverallsystemstateandcommandinformation.Persistoperationallyrequiredinformationlocallyandsendtoarchiveolder
![Page 24: Connected Corridors: I-210 Pilot Integrated Corridor Management … · 2019-12-17 · PARTNERS FOR ADVANCED TRANSPORTATION TECHNOLOGY INSTITUTE OF TRANSPORTATION STUDIES UNIVERSITY](https://reader034.vdocuments.site/reader034/viewer/2022042115/5e92e86bb87eda30616dae3b/html5/thumbnails/24.jpg)
I-210Pilot:CoreSystemHigh-LevelDesign
19
informationforlongertermstorage.Retrieveinformationstoredupondemand(operationaldataforimmediateretrieval,archiveddataforlater,scheduledretrieval).
4. Secureinformationreceived,processedandstored.5. ProvidedatacommunicationsbetweentheprimarysystemsoftheICMcoresystem.
Inordertofulfilltheseroles,thedatahubprovidesaseriesofdatapipelinestoreceiveinformation,processtheinformation,persisttheinformation,andsecuretheinformation.ThepipelinesuseKafkaandActiveMQmessagingsystemstotransmitinformationthatcanbetappedbypersistenceworkerstopersistinformationtothevariousdatastoreswithinthedatahub,andretrievetheinformationandplacetheinformationbackonthemessagingsystemsfordownstreamconsumption.Externalinterfaces,usingApacheCamel,provideinformationandcommunicationbetweenthedatahub,theDSS,andCMSsystems.
Figure5-1DataHubHighLevelDesign
Thisdesignconsistsofindividualservicesconnectedviamessaging.Itisahighlydecoupleddesignbasedonagroupofindependentservicesconnectedbytwomessagingplatforms.Figure5-1providesageneralizeddiagramthatillustratesthreebasictypesofdatapipelines.Sensordataisgenerallyahighfrequency,largerdatavolumepipelinethathandlesalargernumberofsmallermessages.Theheterogeneousdatapipelinetypehandleslargersizedmessagesatlowermessageprocessingfrequencies.Heterogeneousdataischaracterizedbymultiplesourcesprovidingthesametypeofdatainvaryingdegreesofformatandcontentdifferences.Thehomogeneousdatapipelinesaresimilartotheheterogeneousdatapipelines,butgenerallyarebuiltforasingledatasourcethatprovidesallofa
![Page 25: Connected Corridors: I-210 Pilot Integrated Corridor Management … · 2019-12-17 · PARTNERS FOR ADVANCED TRANSPORTATION TECHNOLOGY INSTITUTE OF TRANSPORTATION STUDIES UNIVERSITY](https://reader034.vdocuments.site/reader034/viewer/2022042115/5e92e86bb87eda30616dae3b/html5/thumbnails/25.jpg)
I-210Pilot:CoreSystemHigh-LevelDesign
20
specifictypeofdata.Notethattheheterogeneousandhomogenousdatapipelinesprovidesomedataintothesensordatapipelinewhensensordataispartofthedatareceivedwithinthesepipelines.
Adescriptionofthedifferenttypesofcomponentsisprovidedbelow:
Reader–Readersarethebeginningofthedatapipeline.Theirroleissimple–toconnectandgatherthedatafromadatasourceusingtheprotocolandformatnativetothatdatasource,andtoplacetheinformationwithinadatamessagingchannelfordownstreamprocessing.Ingeneral,itdoeslittleornoprocessingofthedata,withanyprocessinglimitedtosimpleparsingoflargebatchesintoindividualmessages.
Spark–SparkisanApacheSparkinstancethatprovideshighspeed,highvolume,realtimedataprocessingforsensordata.Itsroleistoprovidedatatransformation,dataquality,dataprocessing,andpredictiveanalyticsforsensordata.Itreceivesdatafromreaders,processesthatdata,andplacesthedataintoKafkamessagetopicsforusebydownstreamprocesses.
Cassandra–CassandraisanApacheCassandrainstancethatprovidesstoragefortimeseriesdata,primarilysensingdata.Cassandraprovideshighspeed,highvolume,clusteredstorageforsensingdata.Thisdataislimitedtooperationaldatastorageandisnotintendedasalong-termdatastore.
Postgres–Postgresisanopensourcerelationaldatabaseinstancethatprovidesstorageforrelationalandgeospatialdata,suchasintersectionsignalplans,rampmeterplans,roadnetworks,assetinventories,organizationalinformation.Itisanoperationaldatastoreandnotintendedasalongtermdatastore.
Persistenceworkers–PersistenceworkersprovidebothstorageandretrievalservicesforbothPostgresandCassandradatastores.Persistenceworkerslistentoassignedmessagetopicsandqueues,bothKafkaandActiveMQ,andstoretheinformationontheappropriatedatastore.Requestscanbesenttoapersistenceworkerviaacommandmessagechanneltoretrievedataandplacethatdataonaspecifiedmessagetopicorqueue.
Processors–Processorsareusedinthehomogeneousandheterogeneousdatapipelinestoprocessdatareceivedfromthevariousdatasources.Processorsprovidedatatransformation,quality,anddataprocessingservicesfordatareceivedfromthesesources.
Mongo–MongoisaninstanceofaMongoDBdatabaseandisusedwithintheprocesstotransformdatareceivedonaheterogeneousdatapipeline.ReadersstorethedatareceivedfromasourceintoMongoDB.ProcessorsthanqueryMongofortheinformationrequiredforprocessinganddownstreamprocessesincludingdecisionsupportandcorridormanagementsystems.MongoDBprovidesanidealplatformforstoringinformationretrievedinanativeformatandfastqueryingofthisinformationwithminimalmaintenancerequiredwhensourceformattingischanged.
Datahublogger(DHlogger)–ThedatahubloggerisaninstanceofGreylogthatcollectsandindexesallloggingforeachindividualcomponentwithinthedatahub,andlistenstomessagingtocapturecontrolandprocessloggingmessages.Itsroleistocapturesystemstateanderrormessagesforsystemstatusandtroubleshootingneeds.
![Page 26: Connected Corridors: I-210 Pilot Integrated Corridor Management … · 2019-12-17 · PARTNERS FOR ADVANCED TRANSPORTATION TECHNOLOGY INSTITUTE OF TRANSPORTATION STUDIES UNIVERSITY](https://reader034.vdocuments.site/reader034/viewer/2022042115/5e92e86bb87eda30616dae3b/html5/thumbnails/26.jpg)
I-210Pilot:CoreSystemHigh-LevelDesign
21
Datahubcontrolgateway(DHcontrolgateway)–ThedatahubcontrolgatewayisanAPIgatewaythatprovidesroutingandmanagementofcontrolandstatusmessagesforthedatahub.ItsroleistoreceivemessagesroutedfromtheinterfacesfortheDSSandCMS,aswellasinternaldatahubcontrolchannels,androutethosetotheappropriatechannelsforthereceivingprocess.Thisprovidesencapsulationofindividualservicesandensuresthatnewservices,servicechanges,andmessagesystemmodificationscanbeaccomplishedwithoutimpactingothersystems,andiskeytothedecouplingofthesystemservicesfortheentireICMsystem.ThisgatewayusesanimplementationofApacheCamel.
Externalinterface/Datagateway–Theexternalinterface/datagatewayencapsulatesthefunctionsofthedatahub,providingasortofswitchboardforusebytheCMSandDSSsystems.CMSandDSSsystemsarenotrequiredtoknowtheinternalsofthedatahubmessaging,butinsteadrelyontheexternalinterface/datagatewaytoprovideprotocolspecificcommunicationportsthatareappropriatefortheCMSandDSSandthentoroutemessagestoandfromthedatahubforthem.Thisexternalinterface/datagatewayusesApachecameltoprovidebothroutingandtranslationservices.Internaldatahubmessages(KafkaorActiveMQ)aretranslatedintotheprotocolsusedbytheCMSandDSS(ActiveMQ,RESTservices,orSOAPservices).CommonTMDDandGTFSmessageformatsareusedinmostcommunications,sonotranslationofdataformatsisrequiredbythegateway.
PeMS–PeMSisastatesystemthatcurrentlyprovidesstatetransportationdataservices.PeMSwillprovidelongtermstorageforallsystemdata,archivingservices,andreportingservicesfornon-realtimedataandreportingneeds.
BusinessIntelligence–BIserviceswillbeprovidedbytheCaltrans’OBIEEimplementation.
5.1. DATASOURCES
ThefollowingdatasourceshavebeendefinedfortheI-210ICMproject:
Source InformationType
System Vendor Product C2CTMDD?
Pasadena Intersectionsignal
PasadenaTMC
McCain Transparity Planned
Duarte Intersectionsignal
CountyTMC KimleyHorn
KITS Planned
Monrovia Intersectionsignal
CountyTMC KimleyHorn
KITS Planned
Arcadia Intersectionsignal
ArcadiaTMC Transcore Transuite Planned
CaltransFWTraffic Loopsensing CaltransATMS Parsons Custom N
![Page 27: Connected Corridors: I-210 Pilot Integrated Corridor Management … · 2019-12-17 · PARTNERS FOR ADVANCED TRANSPORTATION TECHNOLOGY INSTITUTE OF TRANSPORTATION STUDIES UNIVERSITY](https://reader034.vdocuments.site/reader034/viewer/2022042115/5e92e86bb87eda30616dae3b/html5/thumbnails/27.jpg)
I-210Pilot:CoreSystemHigh-LevelDesign
22
CaltransFWRamps Rampmeters CaltransATMS Parsons Custom N
CaltransFWCMS DMS CaltransATMS Parsons Custom N
CorridorTrailblazer DMS Notyetidentified
Custom Planned
CaltransIntersections Intersectionsignal
TSMSS Transcore Transuite Planned
Bluetoothtraffic1 Traveltime Vendor Iteris/? Unknown
Bluetoothtraffic2 Traveltime CountyTMC ? Unknown
Environmentalsensing Environmental ? CalPoly ODS ?
RIITSTransit Transit ? CalPoly ODS ?
RIITSVideo* Video RIITS
CaltransVideo* Video viaRIITS
CaltransFWLaneclosure Lanestatus LCS
CityLaneclosure Lanestatus StateHSRsystem
StateofCA Custom N
LACounty Intersectionsignal
CountyTMC KimleyHorn
KITS Planned
CHPCAD Incident CAD manualinputbyoperator
Caltransincident Incident CaltransATMS Parsons Custom Planned
Goldlinetransit Transit NextBus NextBus
![Page 28: Connected Corridors: I-210 Pilot Integrated Corridor Management … · 2019-12-17 · PARTNERS FOR ADVANCED TRANSPORTATION TECHNOLOGY INSTITUTE OF TRANSPORTATION STUDIES UNIVERSITY](https://reader034.vdocuments.site/reader034/viewer/2022042115/5e92e86bb87eda30616dae3b/html5/thumbnails/28.jpg)
I-210Pilot:CoreSystemHigh-LevelDesign
23
*Videoisnotpassedthroughorstoredwithinthedatahub.
5.2. DATAPIPELINES
5.2.1. SENSINGDATAPIPELINE
ThesensingdatapipelinedesignisprovidedinFigure5-2.Inthisillustration,thedifferentsensorfeedscanbeseenontheleft.Thisillustrationshowsthereaderreceivingdatafromadatasource,placingthatdataonaKafkatopic,SparkconsumingthatdataandprocessingthedatautilizingtheMLLibandStreamingSparklibraries,andthenplacingtheprocesseddataonaKafkatopic.DecisionSupportandCorridorManagementsystemscanaccessthatprocesseddatadirectlyviathedatahubdatainterfaces.SparkalsostoresresultsontheCassandracluster.Persistenceworkerscanbeusedtoretrieveprocesseddatawhenrequested,anddatathatisstoredinPostgres,suchassensorinventories,isavailabletoSparkifneededforprocessingthatmayrequiresuchdata.
Figure5-2SensingDataPipelineDesign
ThedatasourceslistedinSection5.1thatwillbeprocessedusingthistypeofpipelineinclude:
Source InformationType
System DataType C2CTMDD?
Pasadena Intersectionsignal
PasadenaTMC
SignalSensing Planned
![Page 29: Connected Corridors: I-210 Pilot Integrated Corridor Management … · 2019-12-17 · PARTNERS FOR ADVANCED TRANSPORTATION TECHNOLOGY INSTITUTE OF TRANSPORTATION STUDIES UNIVERSITY](https://reader034.vdocuments.site/reader034/viewer/2022042115/5e92e86bb87eda30616dae3b/html5/thumbnails/29.jpg)
I-210Pilot:CoreSystemHigh-LevelDesign
24
Duarte Intersectionsignal
CountyTMC SignalSensing Planned
Monrovia Intersectionsignal
CountyTMC SignalSensing Planned
Arcadia Intersectionsignal
ArcadiaTMC SignalSensing Planned
CaltransFWTraffic Loopsensing CaltransATMS
FWsensing N
CaltransFWRamps Rampmeters CaltransATMS
Rampsensing N
CaltransIntersections Intersectionsignal
TSMSS SignalSensing Planned
Bluetoothtraffic1 Traveltime Vendor Traveltime Unknown
Bluetoothtraffic2 Traveltime CountyTMC Traveltime Unknown
RIITSEnvironmentalsensing
Environmental ? EnvironmentalSensing
RIITSTransit Transit ? Transitlocation(future)
LACounty Intersectionsignal
CountyTMC SignalSensing Planned
Goldlinetransit Transit NextBus Transitlocation(future)
5.2.2. CALTRANSFREEWAYTRAFFICDATAPIPELINE
TheCaltransFreewaytrafficdatapipelineisasensingdatapipeline.Itconsistsofallofthestandardsensingpipelinecomponents:
• Kafkatopicsfordatatransmission• Readerforretrievingdata• Sparkfordatatransformation,qualitychecks,machinelearning• Cassandraforpersistence
![Page 30: Connected Corridors: I-210 Pilot Integrated Corridor Management … · 2019-12-17 · PARTNERS FOR ADVANCED TRANSPORTATION TECHNOLOGY INSTITUTE OF TRANSPORTATION STUDIES UNIVERSITY](https://reader034.vdocuments.site/reader034/viewer/2022042115/5e92e86bb87eda30616dae3b/html5/thumbnails/30.jpg)
I-210Pilot:CoreSystemHigh-LevelDesign
25
Figure5-3illustratesthisdatapipeline.
Figure5-3FreewaySensingDataPipeline(PeMS)
PeMS30secondfilesareaflatformatdatafile.DataisretrievedbythereaderviaFTP.Thereaderutilizesaconfigurableschedulerthatwillpollfordatafiles.Whenoneormoredatafilesisavailable,thereaderprocessesthefilesintimeorder,withtheoldestfilefirst.Eachrowwithinthefilecontainstheinformationforasinglesensoratasingletimestep.Allsensorsforasingletimesteparecontainedwithinthefile.Thetimeofeachreadingislocatedinthefilename.ThereaderparseseachrowofthefileandcreatesasingleKafkamessageforeachsensorandtimestep,appendingthetimesteptotheinformationpassedwithinthemessage.Uponcompletionofthefileprocessing,astatusmessageforthefile,includingthetimestepandtheKafkapositionofeachmessageissenttoastatustopic.
Uponreceivingthestatusmessage,Sparkprocessesthemessages,completingthefollowingsteps:
1. ThemessagesaretransformedintoaTMDDdatastructure(Note:thisisnotaTMDDformat,andKafkamessagesareJSON,notXML,anditisnotpassedviaaSOAPprotocol.However,thedatastructureisidenticaltoaTMDDstructure.)
2. Aninventorycheckiscompletedforeachtimestep.3. Aflowbalancecheckiscompletedforallsensorson4and6hourrollingtimewindowsanda24
hourstatictimewindow.4. Ademandpredictioniscompletedforthenexthourforeachfreewayramp.
SparkstorestheresultofitsprocessinginCassandraandpassesthesensinginformationtotheDH_pems_processedtopic.
5.2.3. HETEROGENEOUSDATAPIPELINE
TheheterogeneousdatapipelinedesignisprovidedinFigure5-4.Inthisillustration,thedifferentintersectionsignalfeedscanbeseenontheleft.
ThisillustrationshowsthereaderreceivingdatafromadatasourceandinsertingthatdatainarawmessageformatintoMongoDB.Thereaderretrievesdatafromthedatasourceusingthedatasource’s
![Page 31: Connected Corridors: I-210 Pilot Integrated Corridor Management … · 2019-12-17 · PARTNERS FOR ADVANCED TRANSPORTATION TECHNOLOGY INSTITUTE OF TRANSPORTATION STUDIES UNIVERSITY](https://reader034.vdocuments.site/reader034/viewer/2022042115/5e92e86bb87eda30616dae3b/html5/thumbnails/31.jpg)
I-210Pilot:CoreSystemHigh-LevelDesign
26
nativeprotocolsandformats.ItisdesiredthattheformatbeTMDDformat,andtheprotocolisastandardSOAPprotocol,eitherarequestreceiveorsubscriptionmethodasspecifiedintheTMDDspecification.ThereadertransformsthemessagefromaSOAP/XMLformattoaJSONformatpriortoinsertingintoMongo.
Amessageissentfromthereadertotheprocessortoinformtheprocessoroftheavailabilityofnewdata.
TheprocessorthenqueriesMongotoobtainadesireddocumentwiththespecificattributesrequiredtomakeaTMDDmessagecontainingastandardizedsetofinformationrequiredfordownstreamprocessing.Theprocessoralsoutilizesastandarddictionaryofdatavaluessothatdatafromdifferentsourcesthatmaybedifferentinspecificdefinition,butrepresentthesamevaluearestandardizedfordownstreamconsumption.Forexample,ifonesourceabbreviatesboulevardasBlvd,andaseconddoesnotabbreviatetheword,theprocessorwillstandardizetoasingledefinitionthatcanbeunderstoodbythedownstreamprocesses.Theprocessoralsoconductsdataqualitychecksandanyotherprocessingrequired.
OncetheTMDDtransformation,qualitychecks,andanyprocessingiscompleted,theprocessorconstructsthefinalTMDDstructuredJSONmessageandplacesitontheoutgoingtopic.Persistenceworkerslisteningtothetopicpersisttheinformationforlateranalysisorreplay,andtheCMSandDSSsystemsmayconsumethemessagedirectlyfromtheprocesseddatatopic.
Forintersectionsthatprovideintersectionsensing,theprocessorwillretrievethesensinginformationandsenditviaaseparatetopicforfurtherprocessingwithinthesensing/Sparkstream.
Figure5-4HeterogeneousDataPipeline
![Page 32: Connected Corridors: I-210 Pilot Integrated Corridor Management … · 2019-12-17 · PARTNERS FOR ADVANCED TRANSPORTATION TECHNOLOGY INSTITUTE OF TRANSPORTATION STUDIES UNIVERSITY](https://reader034.vdocuments.site/reader034/viewer/2022042115/5e92e86bb87eda30616dae3b/html5/thumbnails/32.jpg)
I-210Pilot:CoreSystemHigh-LevelDesign
27
5.2.4. HOMOGENOUSDATAPIPELINE
ThehomogenousdatapipelinedesignisprovidedinFigure5-5.Inthisillustration,arampmeterinventoryandstatesourceisusedastheexamplefeedontheleft.Thehomogenousdatapipelineisidenticaltotheheterogeneousdatapipelinewithonlyonedifference.Homogenousdatapipelinesareusedwhenthereisonlyasingledatasourceforthetypeofinformationbeingprocessed.Whenthisoccurs,thereisnoneedtostorethedocumentsreceivedina“schema-less”database,sincetherearenotmultipledatastandardsandimplementationstoaccommodate.Becauseofthis,thehomogenousdatasourceremovestheMongoDBinstallationfromthepipeline,andthereaderandprocessordirectlycommunicateforanydatatransformation,qualitychecks,orprocessingrequired.
Figure5-5HomogenousDataPipeline
5.2.5. PIPELINECONTROL
Allpipelines,regardlessofpipelinetype,areprovidedwiththesamebasecontrolset.Thecontrolsavailableforeachpipelineinclude:
1. Startpipelineprocessing2. Stoppipelineprocessing3. Pipelinestatus4. Replaydata
Allpipelinecontrolrequestsarehandledbythedatahubcontrolgateway.Requestsreceivedbythegatewayareroutedtoanappropriateendpoint.
Startpipelineprocessing–thestartpipelineprocessingwillberoutedviathedatahubcontrolgatewaytotheappropriatereaderwithinapipeline.Therequestwillresultinthesepossibleoutcomes:
![Page 33: Connected Corridors: I-210 Pilot Integrated Corridor Management … · 2019-12-17 · PARTNERS FOR ADVANCED TRANSPORTATION TECHNOLOGY INSTITUTE OF TRANSPORTATION STUDIES UNIVERSITY](https://reader034.vdocuments.site/reader034/viewer/2022042115/5e92e86bb87eda30616dae3b/html5/thumbnails/33.jpg)
I-210Pilot:CoreSystemHigh-LevelDesign
28
1. Ifthepipelineiscurrentlynotstarted,thereaderwillbeginprocessingdataandinsertingitintothepipeline.Thedataprocessedwillbebasedontheconfigurationofthespecificreader.
2. Ifthepipelineisalreadystarted,anerrorwillbereturnedfromthereaderindicatingtheprocessisalreadystarted.
3. Ifthepipelinecannotbestartedforanyreason,anerrorwillbereturned,fromthereaderifpossible,orfromthedatahubcontrolgatewayifnomessageisreceivedfromthereaderorifnoreaderinstanceexists.
4. Anothererror,suchaspipelinedoesnotexistorthattherequestisnotproperlyformatted.
Stoppipelineprocessing–thestoppipelineprocessingwillberoutedviathedatahubcontrolgatewaytotheappropriatereaderwithinapipeline.Therequestwillresultinthesepossibleoutcomes:
1. Ifthepipelineiscurrentlyrunning,thereaderwillstopprocessingdataandwillcompleteinsertionofanycurrentlyprocessingdataintothepipeline.
2. Ifthepipelineisnotcurrentlyrunning,anerrorwillbereturnedfromthereaderindicatingtheprocessisalreadystopped.
3. Ifthepipelinecannotbestoppedforanyreason,anerrorwillbereturned,fromthereaderifpossible,orfromthedatahubcontrolgatewayifnomessageisreceivedfromthereader.
4. Anothererror,suchaspipelinedoesnotexistorthattherequestisnotproperlyformatted.
Pipelinestatus–thepipelinestatusrequestisarequestforasubscriptiontopipelinestatusmessagesthatareproducedatalltimesbyeachprocessingpointwithinapipeline,includingreaders,spark,processors,messagingsystems,andworkers.Thisincludestimesforwhichthepipelineisnotcurrentlyrunning,butthecomponentsarerunning(forinstancewhenthepipelineiscurrentlystopped,butavailableforprocessing).Allpipelinecomponentsprovideataminimumaheartbeatindicatingtheyarecurrentlyupandrunning,evenifnodataisbeingprocessed.
Foreachpipelinestatusrequest,thedatahubcontrolgatewaywillprovideachannelforasubscriptiontoakafkatopiccontainingallstatusmessagesforaspecificpipeline.Pipelinestatusrequestsshallalwaysincludewithintheresponsethetopicidentifier,andmostrecentkafkamessageoffset.Requestersmaythenconnecttothekafkachannelprovidedbythedatahubcontrolgatewayandsetupaconsumerforstatusmessages.
Thefollowingresponsesarepossiblewhenrequestingapipelinestatus:
1. Aresponsewithakafkatopicidentifierandaddress,alongwiththecurrentkafkamessageoffsetandcurrenttime.
2. Ifthestatusmessagesarenotcurrentlyavailable,anerrorwillbereturned.3. Anothererror,suchaspipelinedoesnotexistorthattherequestisnotproperlyformatted.
![Page 34: Connected Corridors: I-210 Pilot Integrated Corridor Management … · 2019-12-17 · PARTNERS FOR ADVANCED TRANSPORTATION TECHNOLOGY INSTITUTE OF TRANSPORTATION STUDIES UNIVERSITY](https://reader034.vdocuments.site/reader034/viewer/2022042115/5e92e86bb87eda30616dae3b/html5/thumbnails/34.jpg)
I-210Pilot:CoreSystemHigh-LevelDesign
29
Figure5-6StreamingPrimaryControlLayer
5.2.6. PIPELINESTATUSANDLOGGING
Allpipelinecomponentsprovideloggingandstatusinformation,regardlessofpipelinetypeorcomponent.LoggingincludestheloggingattheEC2orapplicationlevelthatisprovidedtotheEC2logsattheOSlevelandispublishedtotheGrayloginstancewithinthedatahub.StatusincludesstatusmessagespublishedtostatusKafkatopics,againcollectedbythedatahubGrayloginstance.
Loggingshallbeconfigurableandsetoncomponentstartandshallincludeataminimumthefollowinglogginglevels:
1. All2. Debug3. Fatal4. Error5. Warn6. Info
Logginglevelwillbesetuponcomponentinitializationwhenitisstarted.Ingeneral,duringnormaloperation,thefollowinglevelsshallreport–Error,Fatal,Warn,andInfo.TheinformationgeneratedbyloggingshallonlybeavailablethrougheitherinspectionoftheEC2instancewheretheyweregeneratedandviathedatahubGrayloginstance.
StatusinformationisproducedbythecomponentsandpublishedtoapipelinestatusKafkatopic.Thisinformationisavailabletoexternalconsumersviathedatahubcontrolgateway(usingadatahubstatusrequest)orthedatahubGrayloginstance.Datapipelinestatusmessagesshallbeprovidedforthefollowingataminimum:
![Page 35: Connected Corridors: I-210 Pilot Integrated Corridor Management … · 2019-12-17 · PARTNERS FOR ADVANCED TRANSPORTATION TECHNOLOGY INSTITUTE OF TRANSPORTATION STUDIES UNIVERSITY](https://reader034.vdocuments.site/reader034/viewer/2022042115/5e92e86bb87eda30616dae3b/html5/thumbnails/35.jpg)
I-210Pilot:CoreSystemHigh-LevelDesign
30
1. Pipelinecomponentheartbeat–astatusmessageindicatingthecomponentisrunningsentataregularinterval.
2. Pipelineerrormessage–astatusmessagesentindicatingthatanerrorhasoccurred.Generally,thiswillincludeFatalorErrorlogmessages.
3. Pipelinewarningmessage–astatusmessagesentprovidingawarningindicatingpotentialissuesinoperation.Generally,thiswillincludeallWarnloggingmessages.
4. Pipelineprocessingmessage–thisincludesmessagesindicatingprocessingofasetofdata,oftenatatimesteporfilelevel.Forinstance,inaTMDDsubscription,aprocessingmessagewouldbesetfromeachpipelinecomponentindicatingwhentheinformationfromamessageresultingfromthesubscriptionhasbeenprocessed.Itisnotguaranteedthatdifferentpipelineprocesseswillprocessinformationatthesamegranularity,however,eachstatusmessagewillprovidethecorridorentityidentifiersandtheoriginalmessageidentifierfromtheoriginatingsystemthatstartedthedataprocessing.
Pipelinecomponentsmayincludeotherstatusmessagesasneeded.
Figure5-7Status/LoggingLayer
5.2.7. CORRIDORMANAGEMENTSYSTEM-DECISIONSUPPORTSYSTEM(CMS-DSS)PIPELINE
TheCorridorManagementSystem–DecisionSupportSystem(CMS-DSS)PipelineprovidesthecommunicationbetweentheDecisionSupportSystem,DataHubandtotheCorridorManagementSystem:
• Trafficestimationresults,includinggeospatialinformationandtrafficmetrics• Trafficprediction,includinggeospatialinformationandtrafficmetrics• Responseplansandresponseplanevaluations• DSSStatus• CMSStatus
![Page 36: Connected Corridors: I-210 Pilot Integrated Corridor Management … · 2019-12-17 · PARTNERS FOR ADVANCED TRANSPORTATION TECHNOLOGY INSTITUTE OF TRANSPORTATION STUDIES UNIVERSITY](https://reader034.vdocuments.site/reader034/viewer/2022042115/5e92e86bb87eda30616dae3b/html5/thumbnails/36.jpg)
I-210Pilot:CoreSystemHigh-LevelDesign
31
• DSSControl• Trafficestimationandpredictionscenarioinformation
Inaddition,thepipelineprovidespersistencewithinthedatahubofallDSS–CMScommunications.ThepipelinedoesnotincludeanycommunicationsfromtheCorridorManagementSystemthatdonotinvolvetheDecisionSupportSystem.
AllCMS-DSSpipelinessupportTMDDcompliantSOAPdialogsinaccordancewiththeConnectedCorridorsDataCommunicationSpecification.SOAPendpointsarepresentattheCMSDataHubGatewaytosupporttheseconversationsandareexposedtotheCMS.ThedatafortheseSOAPinterfacesarecommunicatedtotheDSSviatheDataHubanditsKafkaandActiveMQmessagingsystem.AllcommandrequestsandresponsesarealsoprovidedtotheDataHubControlgatewayforanydatahubcontrolactionsrequired.WhileonlytobeprovidedasnecessarytosupportthedifferentCMSvendors,REST,ActiveMQmessaging,andKafkamessagingmayalsobeprovidedasinterfacestotheCMSinsteadofSOAPinterfaces.
AllCMS-DSSpipelinesarealwaysonpipelines.Forfutureuse,datahubswillsupportmultipleCMSandDSSinstances,soallindividualpipelinesarecreatedforeachCMS/DSSpair.AllmessagesbetweenCMSandDSSinstancesshallbecommunicatedthroughonlypipelinesdesignatedforthatspecificCMS-DSSpairandshallincludewithinthemessagesthesourceandtargetsystemidentifiers.
Figure5-8providesanillustrationoftheCMS–DSSpipeline,anditsassociatedprimarydatatopics,including:
• Incidents• ResponsePlans• DecisionResults• StatusRequest• StatusResponse• EstimationRequest• EstimationMetrics• EstimationResults• EstimationNetwork• EstimationScenario• PredictionRequest• PredictionResult• PredictionMetrics• PredictionNetwork• PredictionScenario
![Page 37: Connected Corridors: I-210 Pilot Integrated Corridor Management … · 2019-12-17 · PARTNERS FOR ADVANCED TRANSPORTATION TECHNOLOGY INSTITUTE OF TRANSPORTATION STUDIES UNIVERSITY](https://reader034.vdocuments.site/reader034/viewer/2022042115/5e92e86bb87eda30616dae3b/html5/thumbnails/37.jpg)
I-210Pilot:CoreSystemHigh-LevelDesign
32
Figure5-8CorridorManagement-DecisionSupportSystemPipeline
5.2.7.1. Incident
TheincidentpipelineprovidesconfirmedincidentinformationfromtheCMStotheDSS.ThispipelinemaystartasaRESTrequest,Kafkamessage,ActiveMQmessage,orSOAPincident.AlsonotethatthispipelinerepresentsaTMDDeventclassdialogsetwithbothrequests,responses,andsubscriptions.Subscriptionsarethepreferredmethodofcommunication.TheincidentpipelineincludesanincidentsubscriptionrequesttotheCMS,aswellasthecorrespondingsubscriptionresponses.IncidentrequestsareoriginatedfromtheCMSDataGateway.TheresponsesareprovidedtotheDSSviathedatahubasActiveMQmessages.
5.2.7.2. ResponsePlan
TheresponseplanpipelineprovidesresponseplanstotheCMS.AsthereisnoTMDDdialogforresponseplans,thisisaConnectedCorridorscustomcommunication.Responseplansareprovidedasan
![Page 38: Connected Corridors: I-210 Pilot Integrated Corridor Management … · 2019-12-17 · PARTNERS FOR ADVANCED TRANSPORTATION TECHNOLOGY INSTITUTE OF TRANSPORTATION STUDIES UNIVERSITY](https://reader034.vdocuments.site/reader034/viewer/2022042115/5e92e86bb87eda30616dae3b/html5/thumbnails/38.jpg)
I-210Pilot:CoreSystemHigh-LevelDesign
33
ActiveMQmessageviatheCMSDataGateway.ResponseplansareprovidedinaccordancewiththeConnectedCorridorsDataSpecification,andincludetheresponseplanelements,event/incidentindexidentifier,andassociatedperformanceindicatorsandmetrics.
5.2.7.3. DecisionResult
ThedecisionresultpipelineprovidestheresultsfromtheDSS,providingtheindexoftheselecteddecision,andthekeyvaluepairsforeachresponseplanindicatorandscoreelements.ThedecisionresultisnotaTMDDdialog,andisavailableviaActiveMQmessageviatheCMSDataGateway.
5.2.7.4. StatusRequest
ThestatusrequestisacommunicationchannelforrequestingDSSanddatahubstatus.ItisanActiveMQqueue/message.Requestsmustincludeanidentifierfortherequestingsystem.Aresponsewillincludeanaddressforreceivingstatusmessages.Statusrequestsareessentiallyasubscriptionrequest,withtheresponsesimplyalocationforsubscribingtoacontinuousstreamofstatusmessages.
5.2.7.5. Status
ThedecisionresultpipelineprovidestheresultsfromtheDSS,providingtheindexoftheselecteddecision,andthekeyvaluepairsforeachresponseplanindicatorandscoreelements.ThedecisionresultisnotaTMDDdialog,andisavailableviaActiveMQmessageviatheCMSDataGateway.DatahubandCMSstatusmessagesareprovidedonthesametopicspecifictotheCMS-DSSpair.
5.2.7.6. EstimationRequest
TheestimationrequestpipelineprovidesamethodfortheCMStorequestanestimation.Ifanestimationiscurrentlyrunning,theresponsetotherequestshallindicatetheaddresswhereestimationresultsarepublished.Iftheestimationisnotcurrentlyrunning,theDSSshallstartanestimationandtheresponseshallincludeamessageindicatingtheDSSisbeginninganestimation,andtheaddresswhereestimationresultsarepublished.
5.2.7.7. EstimationResult
Theestimationresultpipelineprovidestherawresultsofanestimationrequest,includingforfreewaystheestimateddensity,flow,andspeedforeachnetworklinkwithinthefreewayestimationnetwork,andforarterials,theestimateddensityforeachnetworklinkwithinthearterialestimationnetwork.
5.2.7.8. EstimationMetrics
Theestimationmetricspipelineprovidesestimationmetricresults,includingtraveltimeandcurrentdelayestimatesforthefreewayandarterialnetworks.
![Page 39: Connected Corridors: I-210 Pilot Integrated Corridor Management … · 2019-12-17 · PARTNERS FOR ADVANCED TRANSPORTATION TECHNOLOGY INSTITUTE OF TRANSPORTATION STUDIES UNIVERSITY](https://reader034.vdocuments.site/reader034/viewer/2022042115/5e92e86bb87eda30616dae3b/html5/thumbnails/39.jpg)
I-210Pilot:CoreSystemHigh-LevelDesign
34
5.2.7.9. EstimationNetwork
TheestimationnetworkpipelineprovidesthecurrentestimationnetworkinaTMDDformat.ThispipelinesupportsTMDDmessaging.
5.2.7.10. EstimationScenario
Theestimationscenariopipelineprovidesthecurrentestimationscenariodatastructure(arterialandfreeway).Itisnotexpectedthatthispipelinewillbeavailableintheinitialsystemdeployment.
5.2.7.11. PredictionRequest
ThepredictionrequestpipelineprovidesamethodfortheCMStorequestaprediction.Theresponsetotherequestshallindicatetheaddresswherepredictionresultsarepublished.Intheinitialversionofthesoftware,predictionsshallonlybeprovidedeitherasaresponsetoanincidentaspartoftheDSSprocess,orshallbeprovidedasareplaytoapreviousprediction.
5.2.7.12. PredictionResult
Thepredictionresultpipelineprovidestherawresultsofapredictionrequest.TODO:determinewhatfromAimsunwillbeprovided.
5.2.7.13. PredictionMetrics
Thepredictionmetricspipelineprovidespredictionmetricresults,includedtraveltimeandcurrentdelaypredictions.
5.2.7.14. PredictionNetwork
ThepredictionnetworkpipelineprovidesthecurrentpredictionnetworkinaTMDDformat.ThispipelinesupportsTMDDmessaging.
5.2.7.15. PredictionScenario
ThepredictionscenariopipelineprovidesthecurrentAimsunpredictionfile.Itisnotexpectedthatthispipelinewillbeavailableintheinitialsystemdeployment.
5.2.8. CMSCOMMANDSTATUSPIPELINE
CommunicationsbetweentheCMSandexternalsystemsorcentersarecapturedbytheDataHub.TheyarealsoprovidedtotheCMScontrolgatewaysothatthedatahubmayrespondtocommandsissuedbytheCMSsystemtoexternalsystems.Figure5-9providesanillustrationoftheCMSCommandStatuspipeline.
![Page 40: Connected Corridors: I-210 Pilot Integrated Corridor Management … · 2019-12-17 · PARTNERS FOR ADVANCED TRANSPORTATION TECHNOLOGY INSTITUTE OF TRANSPORTATION STUDIES UNIVERSITY](https://reader034.vdocuments.site/reader034/viewer/2022042115/5e92e86bb87eda30616dae3b/html5/thumbnails/40.jpg)
I-210Pilot:CoreSystemHigh-LevelDesign
35
Figure5-9CMSCommandStatusPipeline
ThisCMSCommandStatuspipelineisasimpleimplementationofanActiveMQtopicthatisprovidedacopyofallCMScommunicationswithanyexternalcenterorsystem.AllsuchcommunicationsareprovidedviaActiveMQmessagecontainingatextrepresentationofanymessagesentorreceived.Messagesarekeyedwiththeexternalsystemorcentercommunicatedwith,aswellaseithersendorreceivetoindicatewhetherthemessagewassenttotheexternalcenterorsystem(send)orreceivedfromtheexternalcenterorsystem(receive)aswellasthetimeofmessagesentorreceipt,asappropriate.AllmessagesaresavedviapersistenceworkertoaPostgresdatabase.TheDSSdatahubcontrolgatewayalsosubscribestothedatahubCMScommandstatustopicandmayrespondwithspecificcontrolactionswithinthedatahub.
Initially,thestoreforcommandswillbeGraylog,notPostgres.Graylogprovidesauserinterfaceandqueryenginethatallowsuserstointerpretandreviewsystemprocessesthroughthecapturedcommandstatusmessages.
5.3. EXTERNALINTERFACE/DATAGATEWAY
Thedatagatewayprovidesanexternalinterfacetothedatahubforthecorridormanagementsystemandthedecisionsupportsystem.Thepurposeofthisdatagatewayis:
• Providefortwo-waycommunicationbetweentheDataHubandtheDecisionSupportSystem• Providefortwo-waycommunicationbetweentheDataHubandtheCorridorManagement
System• EncapsulatethefunctionsofthedatahubfromotherICMcomponentsystems• Provideasecureinterfacetothedatahub• AllowforthechangefromKafkaorActiveMQprotocolsusedwithinthedatahubtoRESTor
SOAPbasedservicesaswellasKafkaorActiveMQthatmayberequiredbyotherICMcomponentsystems
• AllowforaswitchboardlikefunctionalitythatcanbeconfiguredfordifferentCorridorManagementSystemprovidersorDecisionSupportSystems
• Provideformultipleoutputchannelstopublishinformationtomultipledownstreamconsumers
![Page 41: Connected Corridors: I-210 Pilot Integrated Corridor Management … · 2019-12-17 · PARTNERS FOR ADVANCED TRANSPORTATION TECHNOLOGY INSTITUTE OF TRANSPORTATION STUDIES UNIVERSITY](https://reader034.vdocuments.site/reader034/viewer/2022042115/5e92e86bb87eda30616dae3b/html5/thumbnails/41.jpg)
I-210Pilot:CoreSystemHigh-LevelDesign
36
TheDataHubDataGatewayusesApacheCameltoprovidetheseservices.TwoprimarydesignpatternsareusedasshowninFigure5-10,firstforcommunicationtoexternalActiveMQbrokers,andsecondtoexternalSOAPorRESTwebservicesendpoints.
Figure5-10DataHubDataGateway–ActiveMQandWebServicesDesignPatterns
EachofthetwodesignsusesApachecamelandimplementsaKafkaconsumertoretrievemessagesfromadefinedKafkatopic.Eachalsoprovidesforaprocessorfortransformationandroutingofmessages.ForpassingdatausinganActiveMQtopicorqueuehostedbyanexternalbroker,Camel’sSedacomponentisusedforasynchronousmessagingandmessageflowregulation.ForpassingdataviaSOAPorRESTwebservices,anappropriateSOAPorRESTendpointisprovidedforcommunicationwithanexternalSOAPorRESTendpoint.
NOTE:AllinternaldatacommunicationswithinthedatahubareconductedviaKafkaorActiveMQdatamessagingandJSONformatting.NativedatacommunicationsareimplementedusingTMDDorotherappropriatestandardformatswhenavailable.WhilethesestandardsmaynotimplementJSONformatting,themessageswithinthedatahubthatcontainthisinformationareJSONformatted,retainingthedatastructuresandrelationshipswithintheapplicablestandard.Whenexternallyexposed,thesedatamessagesareserializedbacktotheappropriatedataformatdictatedbythestandardused(TMDDorother).
![Page 42: Connected Corridors: I-210 Pilot Integrated Corridor Management … · 2019-12-17 · PARTNERS FOR ADVANCED TRANSPORTATION TECHNOLOGY INSTITUTE OF TRANSPORTATION STUDIES UNIVERSITY](https://reader034.vdocuments.site/reader034/viewer/2022042115/5e92e86bb87eda30616dae3b/html5/thumbnails/42.jpg)
I-210Pilot:CoreSystemHigh-LevelDesign
37
5.3.1. SUBSCRIPTIONENDPOINTS
Subscribe/Notifypattern(http://www.enterpriseintegrationpatterns.com/patterns/conversation/SubscribeNotify.html)-usealeasetoendifnorenew/continuationmessagereceived?
5.3.2. REQUEST/RESPONSEENDPOINTS
BothActiveMQandKafka–CMSgetsconnectionviacamel–camelmaintainssubscriptiontotopic/queue,ActiveMQusesaLastImageSubscriptionRecoveryPolicysubscriptionrecoverypolicy.Getmostrecentmessage.Kafka–getmostrecentmessage
5.4. DATAHUBCOMMANDGATEWAY
ThedatahubcommandgatewayisaninternaldatahubcomponentthatmanagesallcommandcommunicationandcoordinationactivitiesofthedatahubandorchestratescommunicationandcontrolofallcoreICMsystemcomponents.SincetheDSSandCMSdonotcommunicatedirectly,thedatahuborchestratesworkflowsandcommunicationsbetweenthetwocomponentsandtheinternalcomponentsofthedatahub.Similartothedatahubdatagateway,itusesApacheCameltorouteandprocessCoreICMSystemcommands.Inaddition,itusesaworkflowcomponentdesignedforamicroservicearchitectureusedbyNetflixinitsownproductionenvironmentcalledConductor.Itprovidesthefollowingcapabilities:
• Receiveallcoresystemrequestsforservices• Provideexecutionmanagementofallrequestsfordatahubservices• Provideexecutionmanagementofworkflowprocessingforrequestsinvolvingsimplesinglestep
workflowsorcomplexmulti-stepworkflows• Persistservicerequestsandtheiroutcomes(successorfailure)• Manageinternaloperationsofthedatahub(starting/stoppingservices,restartonfailure,etc)• OrchestrateactionsbetweentheDSS,CMS,anddatahub.
Notethatthedatahubcommandgatewayisaninternaldatahubcomponent.Atnotimearethemessagequeuesandtopicsthatitcommunicateswithdirectlyconnectedtoexternalsystems.Allqueuesandtopicsthatcommunicatewiththiscomponentareinternalqueuesandtopicsconnectedtodatahubcomponentsorthedatahubdatagateway(forcommunicationwithexternalsystems).
![Page 43: Connected Corridors: I-210 Pilot Integrated Corridor Management … · 2019-12-17 · PARTNERS FOR ADVANCED TRANSPORTATION TECHNOLOGY INSTITUTE OF TRANSPORTATION STUDIES UNIVERSITY](https://reader034.vdocuments.site/reader034/viewer/2022042115/5e92e86bb87eda30616dae3b/html5/thumbnails/43.jpg)
I-210Pilot:CoreSystemHigh-LevelDesign
38
Figure5-11DataHubCommandGateway
Thedatahubcommandgatewayconsistsofjustafewprimarycomponenttypes.Theseinclude:
• Conductorandrelatedworkflowcomponents• Camelandrelatedroutingandtransformationcomponents• ActiveMQworkflowstatustopic• ActiveMQworkflowtasktopic• Monitor
5.4.1. CONDUCTOR
Conductorisusedfororchestrationoftheindividualpipelinesandbothinternalandexternalrequestsforservices.ConductorisanopensourceprojectdevelopedbyNetflixtomanagesomeofitsownproductionworkflows,particularlywithamicro-servicearchitecture.ThereareanumberofsimilaritiesbetweentheusecasesforwhichConductorisdesignedandtheorchestrationoftheindividualpipelinesandtherequestsforservicesfromtheDSSandCMS,aswellasorchestrationofthecommunicationsbetweenDSSandCMSthatmakeitwellsuitedforuse.
Conductorisusedinan“as-is”configuration“out-of-the-box”.ThedatahubimplementationusestheRESTinterfacesandin-memorydatabasethatcomeswithConductor.NomodificationsaremadetoConductor.Itprovidesworkflowmanagementandcontrol,withworkflowsdescribedinJSONformat.
![Page 44: Connected Corridors: I-210 Pilot Integrated Corridor Management … · 2019-12-17 · PARTNERS FOR ADVANCED TRANSPORTATION TECHNOLOGY INSTITUTE OF TRANSPORTATION STUDIES UNIVERSITY](https://reader034.vdocuments.site/reader034/viewer/2022042115/5e92e86bb87eda30616dae3b/html5/thumbnails/44.jpg)
I-210Pilot:CoreSystemHigh-LevelDesign
39
5.4.2. CAMEL
ToadaptConductortothedatahubinterfaces,CamelisusedasafacadetotheActiveMQandKafkadatahubinterfaces.CamelprovidestheinterfacebetweenConductor’snativeRESTinterfaceandthedatahub’sActiveMQandKafkadatabuselements.
5.4.3. ACTIVEMQWORKFLOWSTATUSTOPIC
TheActiveMQworkflowstatustopicisusedtoallowindividualdatahubcomponentstoprovideworkflowstatusmessagestoConductor,viatheCamelinterface.Italsoisusedtoprovidegeneralsystemandcomponentstatusinformationforuseinworkflowmanagement.Thedatahubdatagatewayalsoprovidesworkflowandexternalsystemstatus(CMSandDSS)messagesviatheworkflowstatustopic.TheCMSandDSSsystemsareneverrequired(orallowed)todirectlymessageviatheWorkflowstatustopic,butinstead,theircommunicationsaremanagedbythedatahubdatagateway.
5.4.4. ACTIVEMQWORKFLOWTASKTOPIC
TheActiveMQworkflowtasktopicishowworkflowtasksaredistributedfromConductortotheindividualdatahubcomponents,ortheDSSandCMSsystems(viathedatahubdatagateway).Aswiththeworkflowstatustopic,DSSandCMSsystemsareneverdirectlyallowedtocommunicatewiththeworkflowtasktopic.Instead,thedatahubdatagatewaymanagesthedistributionoftasksandtheresultingresponsesviatheDSSandCMSAPIs.
5.4.5. MONITOR
Themonitorlistenstotheworkflowstatustopicforsystemandcomponentstatusmessages,alwaysprovidingdatahub,DSS,andCMSstateandstatusinformationtotheConductorworkflowmanager.ThisallowsConductortorespondtodegradedsystemstate,eitherbydisablingspecificworkflowsorstartingotherworkflowstorespondtosystemcomponentfailures.
6. DECISIONSUPPORTSYSTEMDESIGN
Thedecisionsupportsystem’s(DSS)primaryrolesinclude:
• Provideresponseplansfollowingreceiptofaconfirmedincident• Evaluateresponseplansandrankthembasedonuserdefinedcriteria• ProvideoneormorerecommendationstocorridoroperatorsandstakeholdersviatheCorridor
ManagementSystem,whethertoimplementaresponseplanandwhichresponseplantoimplementforeachreceivedconfirmedincident
• Evaluateimplementedresponseplaneffectivenessandrecommendnewresponseplanswhenappropriate
• ProvideresponseplanevaluationresultsforcorridoroperatorsreviewviatheCorridorManagementSystem
![Page 45: Connected Corridors: I-210 Pilot Integrated Corridor Management … · 2019-12-17 · PARTNERS FOR ADVANCED TRANSPORTATION TECHNOLOGY INSTITUTE OF TRANSPORTATION STUDIES UNIVERSITY](https://reader034.vdocuments.site/reader034/viewer/2022042115/5e92e86bb87eda30616dae3b/html5/thumbnails/45.jpg)
I-210Pilot:CoreSystemHigh-LevelDesign
40
TheDSSaccomplishesthiswiththreeprimarycomponents:
• ResponsePlanManagement• RulesEngine• Modeling
Theresponseplanmanagementcomponentreceivesincidentinformationandcoordinatesthedevelopmentandevaluationofresponseplans.Therulesengineprovidesconfigurablelogic/rulesforthedecisiontocreateresponseplans,responseplancreation,andresponseplanranking.Themodelingcomponentprovidestrafficestimationandpredictioncapabilitiestosupportresponseplancreation,evaluation,andranking.
6.1. DSSHIGHLEVELDESIGN
TheDSS’sthreeprimarycomponentsaredescribedinFigure6-1.TheDSSisoneofthethreeprimaryICMsystemsubsystems,andcommunicateswiththeICMsystemviathedatahub’sdatagateway.
Figure6-1DSSArchitecture
Allcorridorinformationisreceivedfromthedatahubdatagateway.Themodelingandresponseplanmanagementcomponentsreceiveallcorridordataandsystemstatusviaaseriesofdatachannelsavailableonthedatahubdatagateway.
![Page 46: Connected Corridors: I-210 Pilot Integrated Corridor Management … · 2019-12-17 · PARTNERS FOR ADVANCED TRANSPORTATION TECHNOLOGY INSTITUTE OF TRANSPORTATION STUDIES UNIVERSITY](https://reader034.vdocuments.site/reader034/viewer/2022042115/5e92e86bb87eda30616dae3b/html5/thumbnails/46.jpg)
I-210Pilot:CoreSystemHigh-LevelDesign
41
AllcommandstotheDSSandrequestsforDSSstatusarereceivedviaasetofcommandchannelsexposedonthedatahubdatagateway.AllcommandrequestsfromentitiesexternaltotheDSSarereceivedbytheresponseplanmanagementcomponent.Requeststothemodelingcomponentareoriginatedbytheresponseplanmanagementcomponent.DirectrequeststothemodelingcomponentfromsystemsexternaltotheDSSarenotallowed,andmustbemanagedviathecommandsexposedbytheresponseplanmanagementcomponent.
AllcommunicationswiththeDSSviatheDataHubDataGatewayaremanagedviaActiveMQmessaging.CommunicationsbetweentheResponsePlanManagementandModelingareenabledviaRESTorActiveMQmessaging.TheresponseplanmanagementandrulesenginearetightlycoupledviatheirjavabasedAPIs.
OutputfromtheDSSconsistsprimarilyofresponseplansandtheirevaluationsfromtheresponseplanmanagementsystem,andestimationandpredictionresultsfromthemodelingsystem.
Figure6-2DecisionSupportSystemHighLevelDesign
Figure6-2providesaslightlymoredetailedviewoftheDSS.
6.2. RESPONSEPLANMANAGEMENT
Responseplanmanagementprovidesfunctionsforthedevelopmentandselectionofresponseplans,includingcoordinationoftheactionsoftheresponseplanmanagement,rulesengine,andthemodelingcomponentsoftheDSS.
Theresponseplanmanagementcomponentisprimarilyaneventbased,asynchronousservicethatlistensforaconfirmedincidentfromthedatahub,anduponreceipt,orchestratesactionsbetweenitself,therulesengine,andmodelingcomponentstodetermineifaresponseplanmightpositivelyimpacttrafficconditionsandifso,developseveralresponseplans,evaluatethem,andrecommenda
![Page 47: Connected Corridors: I-210 Pilot Integrated Corridor Management … · 2019-12-17 · PARTNERS FOR ADVANCED TRANSPORTATION TECHNOLOGY INSTITUTE OF TRANSPORTATION STUDIES UNIVERSITY](https://reader034.vdocuments.site/reader034/viewer/2022042115/5e92e86bb87eda30616dae3b/html5/thumbnails/47.jpg)
I-210Pilot:CoreSystemHigh-LevelDesign
42
responseplanforimplementation.Theresponseplanmanagementcomponentalsohasalimitedselectionofmethodsimplementedsothatotherspecificrequestscanbemadeviathedatahub.
Theresponseplanmanagementalsomonitorsthedecisionsupportsystemanditscomponents.IthaslimitedcapabilitiestoperformspecificstartupactivitiesrequiredforDSSoperation,monitoreachDSScomponent,andperformlimitedactionsshouldastartupactivityfail(forinstance,ifthemodelingestimationprocessshouldstop,theresponseplanmanagementcomponentshallattempttorestarttheestimationprocess).
Figure6-3ResponsePlanManagementHighLevelDesign
ThecircledareaofFigure6-3illustratesthefundamentalcomponentsoftheresponseplanmanagementcomponent.
Ontheleftoftheresponseplanmanagementcomponentinthefigureisthedatahubinterface.ThisisanActiveMQbasedinterfacethatsubscribestothedatahubdatainterfaceandprovidesassetinformation,routeinformation,andincidentinformationtotheresponseplanmanagementcomponent.Thisinformationismaintainedinalocaldatastore.ItalsolistensforspecificrequestssenttotheDSSviathedatahub.
Theresponsemanager,uponreceiptofaspecificrequestorincidentmanagesthespecificworkflowsrequiredtorespondtotherequestortoprovidearesponseplanrecommendation.Thisincludesrequestingdecisionsandresultsfromtherulesengine,informationregardingcurrenttrafficstatefromthemodelingsystem,orspecificpredictionsforresponseplansreceivedfromtherulesengine.Theresponseplanmanageralsoassemblesthecompleteresponseplan,andprovidesitasaresponsetothedatahubforstorageanddistributiontotheCMS.
![Page 48: Connected Corridors: I-210 Pilot Integrated Corridor Management … · 2019-12-17 · PARTNERS FOR ADVANCED TRANSPORTATION TECHNOLOGY INSTITUTE OF TRANSPORTATION STUDIES UNIVERSITY](https://reader034.vdocuments.site/reader034/viewer/2022042115/5e92e86bb87eda30616dae3b/html5/thumbnails/48.jpg)
I-210Pilot:CoreSystemHigh-LevelDesign
43
6.3. RULESENGINE
TherulesengineisacombinationofjavacomponentsandtheopensourceDroolsrulesenginedesignedtorespondtoalimitednumberofquestionsthatcanbeaskedbytheresponseplanmanager.Theselimitedquestionsinclude:
• Doesthecurrentconfirmedincident,withtheresultsofa“do-nothing”prediction,requiredevelopmentandevaluationofresponseplans?
• Whatresponseplansshouldbeevaluated?Createthoseresponseplansfromalimitedsetofresponseplancomponents.
• Whichresponseplan,giventheresponseplansdevelopedandtheirrespectivepredictions,shouldberecommended,listedinarankedorder?
TherulesengineisaJavalibrarythatisincludedintheresponseplanmanagercomponentwithadefinedapispecifictoansweringthesedefinedquestions,giventhecurrentstateofthecorridorassets,definedlimitswithinasetofuserspecifiedrules(suchas“donotusethisroutebetweenthehoursof3and5pm”),currenttrafficstate,currenttime,andincidentinformation.
Itisintendedthatthefinaluserinterfacefordeveloping,editing,testing,andevaluatingrulesandtheirperformancewillbeintheCMS.However,initially,thiswillbelimitedtorequiringITsupporttoassistwithrulesdevelopment,anduploadingofspreadsheetstoaknownlocationwhererulescanbeaddedorupdated.
6.4. MODELING
Themodelingsystemprovidestwoprimaryservices;trafficstateestimationandtrafficprediction.
6.4.1. MODELINGTECHNIQUES
Threetrafficmodelsareusedwithinthemodelingsystem.Theseinclude:
• AcustomfreewaytrafficestimationmodeldevelopedbyUCBerkeleyusingacelltransmissionmodel(CTM)method.
• AcustomarterialtrafficestimationmodeldevelopedbyUCBerkeleyusinganintersectionqueuemodelmethod.
• AcommercialpredictionmodelusingTSS’sAimsunmodelingsystemandamicro-modelingbehavioralmodel.
![Page 49: Connected Corridors: I-210 Pilot Integrated Corridor Management … · 2019-12-17 · PARTNERS FOR ADVANCED TRANSPORTATION TECHNOLOGY INSTITUTE OF TRANSPORTATION STUDIES UNIVERSITY](https://reader034.vdocuments.site/reader034/viewer/2022042115/5e92e86bb87eda30616dae3b/html5/thumbnails/49.jpg)
I-210Pilot:CoreSystemHigh-LevelDesign
44
Figure6-4-ModelingComponentDesign
Thesystemiscomposedofseveralprimarysubsystems–• Datamanagement(inblue)–receivesandprocessestrafficandtrafficinfrastructuredataso
thatitcanbeconsumedbythemodelingsoftware.• Estimationmodelingsystem(inpink)–buildsandrunsestimations,producesoutputof
estimatedtrafficstate.• Predictionmodelingsystem(inorange)–buildsandrunspredictedtrafficstate,inparticular
predictedresponseplanperformance,includingperformancemetrics.• Calculation/metrics(inlightorange)–consumesestimatedtrafficstateandproducescurrent
trafficandnetworkperformancemeasures.• ProjectManager(inbrown)–providesoverallsystemorchestration,cloudscalingandEC2
management,workflow,andexternalRESTinterfaces.Auserinterfaceisavailableforestimationmodeldevelopment.ThisiscurrentlyusedforinternaltestingandQA.
• Persistence–(green/gray)–persiststraffic/trafficinfrastructuredata,modelandmodelbuildingdefinitions,rawandprocessedresults.
CommunicationwithexternalsystemsandthemodelingsystemisviaaRESTAPIexposedbytheProjectManager.DatafeedsfromthedatahubareprovideddataviaActiveMQdatatopics.InternalcomponentcommunicationbetweenmodelingsystemcomponentsisviaActiveMQtopics(data)orqueues(command).
![Page 50: Connected Corridors: I-210 Pilot Integrated Corridor Management … · 2019-12-17 · PARTNERS FOR ADVANCED TRANSPORTATION TECHNOLOGY INSTITUTE OF TRANSPORTATION STUDIES UNIVERSITY](https://reader034.vdocuments.site/reader034/viewer/2022042115/5e92e86bb87eda30616dae3b/html5/thumbnails/50.jpg)
I-210Pilot:CoreSystemHigh-LevelDesign
45
Technologystack:• PrimarycomponentsarebuiltusingJavawiththeSpringframework.Hibernateisusedwith
componentsrequiringPostgresaccess.• PersistenceisbuiltwithJavabasedpersistenceworkers(bothforstoreandretrieveoperations)
andPostgresv9.6.2orCassandrav3.10forpersistence.• MessagingviaActiveMQv5.14.5• ProjectManagerRESTserviceshostedonTomcatv8.5.15• LogManagementviaGraylog• AmazonWebServices,includingthefollowingAWSservices
o EC2o RDS(Postgres)o S3o VPCo IAMo Otherservicesareusedforcoderepository,build,anddeploymentservices
Platform/ApplicationServers/OS• MostanyversionofLinuxOSisacceptable,Ubuntu14.04isthecurrentstandard• AmazonWebServices• Tomcatv8.5.15
GeneralArchitecturalApproach:
Theapproachusedhasbeentodevelopspecificapplicationcomponentstargetedtospecifictasks,connectthemtoeachotherviamessagingorRESTservicesdependingupontheirapplicationrole(backendprocessingorservicedelivery),andprovidescalingcomponentsthatalloweachindividualcomponenttodetectload(CPU/threadutilizationanditsfeedingqueuestate)andscalethenumberofinstancesoftheresourcetypebasedondemand.
Primarycomponentdescriptions:
ModelEngine–Therearetwoversionsofthemodelengine.Thefirstisusedforfreewayestimation.ThefreewaymodelengineisthecorecomponentofthesystemthatimplementstheCTMalgorithm.Thisimplementationisasimulationusingauserdefinedroadnetwork,accompanyinginfrastructureelementsincludingrampmetersandsensors,trafficdemand,trafficbehaviorinformation(splitratiosatintersectionsandfreewayramps),androad/trafficcharacteristicsrequiredbytheCTMalgorithm(fundamentaldiagrams).Thesimulationcanbeusedtoprovideestimationsofcurrentstateorpredictionsoffuturestate,butisusedonlyforfreewaycurrenttrafficstatewithintheICMsystem.
Thesecondversionofthemodelengineisusedforarterialestimation.Thisarterialmodelengineusesanalgorithmusingintersectionsensingandsignaltimingandcycleinformationtoproduceanestimatedtrafficstateofthecorridorarterials.
Bothtypesofmodelenginereceivearequestforamodelruncontainingthescenariotoberunandthedesiredconfigurationparameters.Themodelenginethenrunsthemodelandoutputsthelinkstate(trafficdensityandspeed)foreachroadnetworklink.
![Page 51: Connected Corridors: I-210 Pilot Integrated Corridor Management … · 2019-12-17 · PARTNERS FOR ADVANCED TRANSPORTATION TECHNOLOGY INSTITUTE OF TRANSPORTATION STUDIES UNIVERSITY](https://reader034.vdocuments.site/reader034/viewer/2022042115/5e92e86bb87eda30616dae3b/html5/thumbnails/51.jpg)
I-210Pilot:CoreSystemHigh-LevelDesign
46
ThemodelengineisamultithreadedJavaapplication,buttheprimaryworkcurrentlyremainssingle-threaded.
Predictionengine–ThepredictionengineisanimplementationofTSS’sAimsunsoftware.ItincludestheAimsunsimulationsoftwareaswellasacomponentthatbuildssimulationmodelsforuseinthepredictionworkflow.Thesimulationbuildcomponentcombinesabasesimulationmodelwithdatafromthedatahubindicatingcurrentcorridorassetstate(signals,ramps,signs,etc),currenttrafficstatefromthefreewayandarterialmodelengines,andtheresponseplansfromtheresponseplanmanagementsystemtocreateasetofoneormoremodelsthatcanbeusedtoevaluatearesponseplan.Italsoprovidesthepredictionmetricsrequiredforpresentationtotheuserandtherulesengineforresponseplanrecommendationandselection.
ProjectManager/ProjectManagerApp-Theprojectmanagerappallowsexternalsystemstointeractwiththemodelingsystem.ProjectmanagerappprovidesRESTservicesandmessagingtoinitiateactionsbytherestofthesystem.TheprojectmanagerappisservedviaaTomcatapplicationserver.
ScenarioBuilder-Thescenariobuilderisusedtocreateascenariotoberun.Itisrarelyused,butcanbeusedwhencorridorinfrastructureelementsaremodified,suchasarampmeter.Itisaback-endcomponentthatreceivesmodelcomponents(roadnetworks,intersectionsignals,rampmeters,trafficdemandandsplitsets,etc)fromtheprojectmanagerappandbuildsamodelsuitableforrunninginthefreewaymodelengine.
Calculators-Thecalculatorsareusedtotakethemodelenginelinkstatedataandprocessitforconsumptioneitherasspecificmetricsforanalysis,orasvisualizationinformationfortheCMS.Eachcalculatorjobissinglethreadedandisdesignedforaspecificcalculation.
PersistenceWorker-Persistenceworkersarecomponentsthataredesignedtoeithertakedatafromaqueueortopic(modelenginedata,model/scenarioinformation,orcalculatoroutput)andpersistitinoneofthedatastores(PostgresorCassandra)or,uponrequestretrievethedatafromoneofthedatastoresandplaceinontheappropriatequeueortopicforconsumptionbydownstreamprocesses.
ReadersandProcessors-Thesecomponentsareusedtoretrieveoracceptpusheddatafromthedatahubandplaceitintoaqueueortopicforconsumption(reader)andtoretrievedatafromaqueueortopicandprocessitforconsumptionbythemodel(processor)orpredictionengine.