Closed-Loop Experiments and Consumer Insights: A

Financial Services View Robert J. Kauffman Deputy Director, LARC, and Associate Dean (Research)

School of Information Systems Singapore Management University

Rakesh Menon Senior Vice President Decision Management

Citibank

October 2012

The Financial Services View

•  Customer: Existing, prospects • Offerer: Bank, merchants, and

competitors • Offers: Products and services

from bank, partners, competitors •  Influencer/s: Anything that

influences a customer’s choice (economy, ads, friends, news)

•  Highly connected, complicated web of different networks

•  Unknown effects of influencers •  Temporal and spatial dynamicity • Greatly affected by competition,

tech changes, consumer demand

System Description •  Ecosystem modeling complex

•  What ifs, controls, adaptiveness • Maintaining dynamicity

•  Temporal and geospatial changes • Maintaining consistency

•  Robust design •  Time-to-market: accelerated tests

Analytics Challenges

Ecosystem Entities Industry Ecosystem

Multiple Forms of Financial Services ‘Networks’

•  Consumer and customer social and geolocation networks for the bank

•  Networks of products and services in the marketplace –  Related designs of products –  Relevant families for services

•  Physical facilities network: –  Branches, ATMs, extensions

(Internet and mobile banking), etc. •  Generalization of human social networks for the

financial services industry

5

Desiderata for FS Analytics •  Closed-loop experimentation should be a business value-

driven activity •  Achieve clarity around: what issues, what contexts, what

experiments, what value, what actions •  Think about information value of next iteration relative to

cost in closed-loop experiments •  Targeted application areas for closed-loop experimentation

in financial services – the offers and influencers: –  Customers, accounts, services / products, cards, facility

types, stakeholders (merchants, malls, banks, etc.), locations, macro/micro-markets

•  Useful analogy: product development process – from design to test to market

Business Value Focus

•  Identify business use cases for research •  Potential value for experimentation critical •  Emphasize experimenting with experimentation

–  Iterative process –  Acquire business-relevant, actionable info –  Work to create decision process-changing artifacts

•  Think about information value of next iteration relative to cost in closed-loop experiments

•  Useful analogy: product assessment process – from design to test to market

Retrospective vs. Prospective Closed-Loop Experimentation

•  Bankers need to understand what causes performance outcomes at different times

•  Match closed-loop experimentation to different time periods of managerial inquiry: –  Past: natural experiments and empirical tests –  Present: controlled experiments with treatment

groups; capture of real-time flows of data –  Future: evolving experimental controls and treatment

groups based on closed-loop learning process and refinement of managerial questions

•  Real-time data flows relevant in retail banking?

Observational Micro-Segmentation and Segment-Based Experiments

Source: Marketelligent, 2012

Customer Performance Experiments Increase customer loyalty, usage ad activation through the testing and evaluation of focused targeted and context-aware customer offers. These work like experimental interventions.

Source: Marketelligent, 2012

Closed-Loop Contextual Awareness

The Financial Services View

•  Customer: Existing, prospects • Offerer: Bank, merchants, and

competitors • Offers: Products and services

from bank, partners, competitors •  Influencer/s: Anything that

influences a customer’s choice (economy, ads, friends, news)

•  Highly connected, complicated web of different networks

•  Unknown effects of influencers •  Temporal and spatial dynamicity • Greatly affected by competition,

tech changes, consumer demand

System Description •  Ecosystem modeling complex

•  What ifs, controls, adaptiveness • Maintaining dynamicity

•  Temporal and geospatial changes • Maintaining consistency

•  Robust design •  Time-to-market: accelerated tests

Analytics Challenges

Ecosystem Entities Industry Ecosystem