real-time analytics in financial

1 ©HortonworksInc.2011– 2016.AllRightsReserved

Real-timeAnalyticsinFinancialUseCase,ArchitectureandChallenges

蒋逸峰（しょういつほう／YifengJiang）SolutionsEngineer,Hortonworks@uprushOctober26,2016


TheYAP

MapbyGoogleMaps


http://www.wondermondo.com


Today’sMoney&FinancialService

moneymoney

FinancialService

0110010100 0110010100


EveryFinancialServiceisaBigDataService

Ã FinancialservicesareBIG

– Toobigtofail

Ã Everyfinancialserviceiseventuallyabigdataservice

– Numberoftransactions

– Numberofjobs

– Thirdpartydata


HowBigisBigDatainFinancial?

Ã Millionstobillionstransactionsperday

– Hundredstotensofthousandstransactionspersecond

Ã BigDatainbanking,payment,security,etc.


BigDataUseCaseinFinancial

http://www.forbes.com/sites/bernardmarr/2016/09/09/big-data-in-banking-how-citibank-delivers-real-business-benefits-with-their-data-first-approach/#7759859f75ed


WhyReal-timeAnalyticsinFinancial?

Canyoudetectfraudfrom

millionstobillionstransactionsperday

inreal-time ?

“Thecostsresultingfromtheseanomaliesisfareasiertocorrectifspottedquickly– or

evenbeforeithappens– throughpredictivemodeling.”


最近気になったニュース

http://gendai.ismedia.jp/articles/-/48832

http://mainichi.jp/articles/20161012/k00/00e/040/243000c

10 ©HortonworksInc.2011– 2016.AllRightsReserved https://roboteer-tokyo.com/archives/4415


Real-timeAnalyticsinFinancialUseCase,Architecture&Challenging


ASimpleUseCase– Real-timeSurveillance

Detectabnormal transactionsinStockExchange

Ã Triggeralertif

– Acustomerbuy/sellamountexceeds500MJPYin3minutes

Ã 300Ktransactionspersecond

Ã Abnormalmustbedetectedwithin10s

Alert


Real-time Surveillance Architecture v1

Trading Data(real-time)

Message Bus(Kafka)

Enricher(Storm)

Aggregator(Storm)

Master data, raw & aggregated trade(HBase+Phoenix)

Surveillance Rule Engine

Surveillance Alerts

master data look up

Insert trade(raw & aggregated)

Architecturev1

how?


Real-timeDataIngestion

Ã From transactiondatabase

– ChangeDataCapture(CDC)

– Notpracticalformostfinancialsystem

Ã Fromgatewaysystem

– Receivedatafromgatewaysystem

– SenddatatoKafka(asKafkaproducer)




Message Bus(Kafka)

Enricher(Storm)

Aggregator(Storm)



Surveillance Alerts

master data look up


via CDC or gateway

Architecturev1

overhead?


Real-timeDataLookup

Ã Lowlatencydatastore

– Masterdata

– NoSQLdatabase:HBase (+Phoenix),Redis

Ã UselocalCache

– LRUcacheinStormbolts




Message Bus(Kafka)

Enricher(Storm)

Aggregator(Storm)



Surveillance Alerts

local master data cache

master data look up


via CDC or gateway

Architecturev2– WithCache

exactly-once?

exactly-once?


Exactly-Once?

Messagedeliverysemantics

Ã At-most-once:maylosedatabutnoduplication

Ã At-least-once:nodataloss,butmayduplicate

Ã Exactly-once:nodataloss,noduplication


Exactly-Once!

NOrealexactly-oncemessagedeliveryindistributedsystem

Ã Thereisnosuchthingasexactly-oncedelivery

Ã Exactly-onceisanend-to-endrequirement

But… peoplelikeexactly-once,especiallyinfinancialservicesystem


Exactly-oncesemantics (betterphrase“effectively-once”)withat-least-once+

idempotentoperations

Ã Kafka&Stormguaranteeat-least-once

Ã De-duplicatebyensuringidempotentinyourapplication

Effectively-once


De-duplicationinwindowcomputation

Mostwindowcomputationscanachieveidempotent

Ã Examples:aggregation,counting,etc.

Ã De-duplicatemessagesinthewindow

– Usinglocalin-memorystatestore,e.g.aSetclass

Trading Eventsin Kafka

IDRegistry(local in-memory)

2. lookup trade_id

3. count de-duplicated events

5. output aggregated data

Aggregated Trade Data

Aggregator(Storm)

4. Insert trade_id

1. Pull data in 3m window


De-duplicationinnon-idempotentcomputation

Ã Exactly-onceinnon-idempotentcomputation

– Example:joincontinuousdatastreams

– Globalstatestorerequired:HBase,Redis

– BatchingcanhelpreducenumberofIDlookup.

Ã Exactly-onceisexpensive,avoiditatthebest

Click Logsin Kafka

IDRegistry(external NoSQL)

2. Lookup click_id

5. Output joined click

Joined Click Logs

Joiner(Storm)

4. Insert click_id

1. Pull data continuously

Query Logs

3. Lookup query




Message Bus(Kafka)

Enricher(Storm)

Aggregator(Storm)



Surveillance Alerts

master data look up

IDRegistry look up / insert,de-duplicate in window




via CDC or gateway

Architecturev3– effectively-once

order?


LateMessages

http://www.slideshare.net/HadoopSummit/apache-beam-a-unified-model-for-batch-and-stream-processing-data


HandlingLateMessages

Ã Expectlatemessages

– Streamingapplicationneedstohandleoutoforderevents,e.g.,emitslatemessagestoaspecial

Kafkatopic

Ã Usesourcegeneratedtimestamp

Ã Storm’slatemessagesupportinwindowcomputation(BaseWindowedBolt)

– withTimestampField(StringfieldName)

– withLag(Durationduration)


CanItrustthedata?

Duplications!

Outoforderlatemessages!

Dataloss?


MonitorDataProcessingPipelineQuality

Approachestomonitordatapipelinequality

Ã AuditcompletenessÃ Outputduplicatedandlatemessagestologsforauditing.Ã Defineservicelevelobjective(SLO)ofdataqualityandmonitortheSLO.


DefineDataProcessingPipelineSLO

DesignpracticalSLOforthepipeline

Ã Process99.9999%eventswithinafewsecondsÃ and100%eventswithinafewhours

Ã At-most-oncesemanticsatanypointoftimeÃ Nearexactly-oncesemanticsinnearreal-timeÃ Andexactly-oncesemanticseventually


TheRuleEngine&TheArchitecture

Hundredsofrules

Ã Astocktradingpricejumpup/down>k%andtotalamount>m%inKminutes

Ã SingleATMcashwithdrawal>k%andnumberofATM>minKminutes

Manyoftheserulesfitintothissimplearchitecture!

RuleEngine

✓✗? ? ? ✓

Rulebaseonly?


Architecture-- withPredictiveAnalytics

Real-time Surveillance Architecture — with Predicate Engine


Message Bus(Kafka)

Enricher(Storm)

Aggregator(Storm)



Surveillance Alerts

master data look up

IDRegistry look up / insert,de-duplicate in window




Financial Data Lake

Train Machine Learning Model

(Spark)load ML model

Surveillance Predicate Engine

(Storm)

via CDC or gateway


LifecycleofBigDataAdoptioninFinancialServiceIndustry

2.BusinessIntelligenceDataminingandvisualizationsoftwarethatrevealstrendsandusefulinformation

1.DataPoolingandProcessingConnectdataandcreatestructurebymerging,conditioningstreamsandarchiveddata

3.PredictiveAnalyticsAutomatedanalyticsintegratedintoworkflowthatunlockdatavalueandimproveprofitability

HadoopenabledBigDataPlatform

Customerstypically“StartSmall,ThinkBig”


THANKYOU

real-time analytics in financial

Technology