transport for london - using data to keep london moving

Transport for LondonUsing data to keep London moving

Sriskandarajah SuhothayanAssociate Director / ArchitectWSO2

OVERVIEW

• Based on the presentation presented at– Strata Data Conference, London, 2017.

• Co-presented with :

– Roland MajorEnterprise ArchitectFormerly Transport of London

2

• Introduction of Transport for London• Surface Intelligent Transport System• Conceptual Architecture

• Introduction to WSO2 technologies• Data in Motion - Hack Week

• ‘Live Journey Planner’ prototype• How It’s Done?

• Data Driven Operational Applications• Game Changing Visualizations• Learning Outcomes• What's next in WSO2 Analytics

AGENDA

• 30 million journeys daily• In addition to all road and rail

transport, they look after rivers, assisted travel, taxi and private hireregulation

• They also do much more fromeducation to running the world’slargest out of home advertising estateof its type

• They are 150 years old and a chock fullof heritage and design assets

BREADTH OF TRANSPORT FOR LONDON

SponsorshipOwn the improvement strategy for the Transport for London Road NetworkEngage with a wide range of customers and external stakeholdersDeliver user benefits that are clearly defined and measured

Outcome DeliveryHighways design and engineeringTraffic modelling capabilityIntelligent control of traffic signalsMonitor, analyse and optimise road network performance

Operations24/7 Operation to keep London movingReal-time incident management through LSTCCAssess and coordinate works and schemes to minimise disruption on TfL’s roads

ROAD SPACE MANAGEMENT

SITS will

• provide the capability to unlock significant additional effective capacity onthe road network for the future

• enable and support delivery of a multi-modal approach to transportmanagement by using and allocating existing and new capacity

• enable and support delivery of a Balanced Scorecard approach to transportmanagement by using and allocating existing and new capacity based onlocal modal demands

SURFACE INTELLIGENT TRANSPORT SYSTEM

Key CharacteristicsBringing information and insightcloser to decisions

Locale, map based UITimely, event driven not batchTrusted, consistent and accurate

Public cloud hostedReusable commodity platformsOpen StandardsMiddleware Platform - WSO2

CONCEPTUALARCHITECTURE

Integration

Data Hub

Collaboration and Innovation

Analytics and Visualisation

Data DrivenOperationalApplications

Secure Enclave

• Enabler for Digital Transformation

• 100% Open Source Middleware Platform

• Offices in : Mountain View, New York, London, Sao Paolo, Colombo

• 350+ Customers

• 450 People, 300 Engineers

WSO2

OPEN TECHNOLOGY FOR AGILE DIGITAL BUSINESS

100% open source

September 26-29, 2016Objective : Managing the Capacity of London’s TransportNetwork

• Maximizing capacity on the public transportnetwork

• Maximizing capacity on the roads network• Improving air quality

Datasets• TfL APIs (Realtime and Historical)• SCOOT sensor reading (Realtime)• Passenger flow (Historical)• Air quality (from KCL) (Historical)

Solution• ‘Live Journey Planner’ Prototype

DATA IN MOTION - HACK WEEK

TfL has about 14,000 sensors measuring roads approaching junctionsThis data is currently used by the Real Time control to manage optimizationRequirements to process

780 Million events per dayLatency to data center circa 1 SecondAt data capturing resolution of 250ms scans

TRAFFIC CONTROL SENSORS

JunctionSCOOT Sensor

CHANCE OF GET A SEAT IN THE TRAIN?

Summarized Oyster Card Data

REALTIME TRAFFIC

High Traffic

Low Traffic

Collect SCOOT DataLearning traffic patterns using R

Building Random Forest Classification (it’s 88% accurate for this usecase) !

Exported the model as PMMLUse the model to predict traffic in realtime with WSO2 DAS

LOOKING INTO THE FUTURE

INCIDENT !

Via Local and Waze

WALKING AND CYCLING

Via Local and Waze

ARCHITECTURE

• Realtime data processing pipeline

HOW IT’S DONE ?

Detect Headway and Vehicle Length

Traffic and Flow

Calculation

Integrating Historic

Summarization

Predicting Traffic

Potential Incident Analysis

On raw SCOOT data stream

Sample stream: 1111100000111111100000

define stream ScootStream (scootId string, time long, reading int, seqId long);

from every e1=ScootStream[reading==1], e2=ScootStream[reading==0]+, e3=ScootStream[reading==1]+, e4=ScootStream[reading==0]

select e3[0].seqId - e2[0].seqId as headway, e4.seqId - e3[0].seqId as vehicleLength, ...

insert into DetectorStream;

DETECT HEADWAY AND VEHICLE LENGTH


Traffic and Flow Calculation

Integrating Historic Summarization Predicting Traffic Potential Incident

Analysis

Pattern Matching

On all SCOOT Detectors in London region.

from DetectorStream[str:split(scootId, "-", 0) == ‘London’]#window.time(’1 min’)select count(*)/60 as flow,

avg(vehicleLength)/60 as traffic, 1/avg(headway) as density, ...

group by scootIdinsert into TrafficStream;

The results are mapped to links and presented via APIs for visual representation

TRAFFIC AND FLOW CALCULATION




Analysis

Filtering

Sliding Time Window

Group By Aggregations

Historic data analysis with Apache SparkJoining With Summarized Data@from(table=‘rdbms’, url=‘...’, ...)define table TrafficSummery (scootId string, week int, day string, traffic long);

from TrafficStream as ts join TrafficSummery as tton ts.week == tt.week and ts.day == tt.day

select ts.traffic as currentTraffic, tt.traffic as usualTraffic, ...insert into SummeryTrafficStream;

INTEGRATING WITH HISTORIC SUMMARIZATION




Analysis

Join

Predicting traffic in next 15 minutes

from SummeryTrafficStream#pmml:predict(’wso2das-3.1.0/marbel_model.pmml')

select * insert into PredictedTrafficStream;

PREDICTIONS




Analysis

Predict Function

Increasing trend in traffic hikes

from PredictedTrafficStreamselect currentTraffic - historicTraffic as currentHike,

predictedTraffic – currentTraffic as predictedHike, ...having currentTrafficHike > 0 and predictedTrafficHike > 0insert into TrafficHikeStream;

from every e1=TrafficHikeStream -> e2=TrafficHikeStream[ (e2.currentHick – e1.currentHike)*100.0 / e1.currentHike > 20 and

(e2.predictedHike – e1.predictedHike)*100.0 / e1.predictedHike > 20]with in 15 min insert into PotentialIncidentStream;

POTENTIAL INCIDENT ANALYSIS




Analysis

Pattern Matching

London Works 2• Central Register – a pan London system enabling visibility and management of works and

related activities in London• Traffic Management Act Notifications (TMAN) - A dedicated interface between London

boroughs and TfL enabling the balanced delivery of major schemes and works on the TLRNand SRN

• Forward Planning Tool - An advance planning tool that allows promoters to provide earlyvisibility of road and street works

Data Driven Operational Applications

GAME CHANGING VISUALIZATIONS

Situational Awareness

• Realtime analytics on data provides edge• Keep focus on usability• Use the right tool for the right task• Skills are the biggest hurdle• Bringing information sets together encourages new thinking• Using Agile approaches has transformed outcomes• Removing system fragmentation has a big impact on organization• Flattening delivery structures & small staged initiatives• Platform approach• Early indications of making data easily shared and integrated is improving

decisions

LEARNING POINTS

Available by end ofyear 2017

34

WHATS NEXT IN WSO2 ANALYTICS

• Support for Machine Learning model execution• Online machine learning, PMML, and Java models

• Support to consume and publish data to and from various transports andin multiple data formats• HTTP, Kafka, Thrift, TCP, JMS, MQTT, RabbitMQ, Email• XML, JSON, Text, CSV, Binary

• Support for connecting to various data stores• RDBMs, MongoDB, Cassandra, HBase, Solr

• Incremental long running data aggregation• Without a need of spark cluster

35

SUPPORTED FUNCTIONALITY

• Rich developer tool• With auto completion, simulation and debugging support

• System monitoring support• Business user friendly configurations• Realtime dashboard

• With gadget generation support

36

TOOLING SUPPORT

37

DEVELOPER STUDIO

38

BUSINESS RULES

39

DASHBOARD

• Scalable deployment with Apache Kafka• Exactly once data processing• Zero data loss

• Minimum high available deployment with 2 nodes• Multi data center support• Zero downtime

40

SCALABILITY AND HIGH AVAILABILITY

• Strata Talk on “Transport for London: Using data to keep London moving” https://conferences.oreilly.com/strata/strata-eu-2017/public/schedule/detail/57582

• WSO2 Analytics https://wso2.com/analytics• WSO2 Data Analytics Server https://wso2.com/analytics/features• WSO2 Stream Processor (Will be released on end of 2017)

https://github.com/wso2/product-sp/releases• WSO2 Siddhi https://wso2.github.io/siddhi/• WSO2 Support https://wso2.com/support/

41

USEFUL LINKS

THANK YOU

wso2.com

transport for london - using data to keep london moving

Technology