eharmony @ phoenix con 2016

V I JAY VA N G A PA N D U

@ v i j a y v a ng a p a nd uv i j ay ku marv a ng a p a nd u

W H O A R E W E ?

EH AR M O NY CR EAT E STH E H A PP I ES T ,M O S T PAS S I O NAT EAND M O S T FU L F I L L I NG R EL AT I O NS H I PS *

*ACC OR D I NG TO A R E C E N T ST U DY

4 3 8M A R R I AG E S P E R DAY

150 questions

PersonalityValuesAttributesBeliefs

M ATC H I N G S Y S T E M

Compatibility Matching System®

C OM PAT IB I L I T Y MATC H ING A F F IN I T Y MATC H ING MATC H

D IS T R I B U T I ON

IntellectEnergy

SociabilityAmbition

KindnessCuriosity

HumorSpiritualit

y

U S E R U P DAT E S

M ATC H D E L I V E RY (V 1 )

M A P-S I DE J O I N S( TB ) SC OR I N G

VOL DE M ORT

MATC H DATA S E RV I CE

MATC H I NG S YS TE M

30+ M I LL I ON E VE NT S

65+ M I L L I ON U SE RS30+ B I L L I ON R E C ORD S

VO L D E M O RT ?T H AT N A M E

T H AT N A M ES O U N D S FAM I L I A R

VO L DE M O RT

AUT OPART I T I O N I NG

P LU G GAB L E SE R IAL I ZAT IO N

AU T OR EP L IC AT I O N

KE Y-VA LUE DY NAM O

G O SS I P

U S E R U P DAT E S O ( N ^ 2 )

* USE R UPDAT ED T HE AD DR E SS

N E E D F O R S C A L AB I L I T Y

VOL DE MORT

30+ Million Match Events / Day

30+ Billion Match Records

Millions of user generated Events / Day

Low latency user requests

1 . 4G B / M IN (14 )

N E E D F O R S C A L AB I L I T YG E T M ATC H E S R E S P ON S E T I M E S

DATA STORE NEEDS

QU E R I E S

LOW L AT E N CY

C R U DOP E RAT I ON S F I LT E R I N G

T H R O U G H P UT40+ M I LL I O N

W R I T E S

30+ B I LL I ONR E C OR D S

DATA STORE NEEDS

E AS Y TOM A I NTA I N

C ON S IS T E N CY AVA I L AB LEPA RT I T I ON

T O LE RA N C E

BREAKING CAP ?

Consistency

Availability PartitionTolerance

CA CP

AP

MongoDBHBaseRedis

CassandraDynamoDB

Riak

RDBMSKAFKA

L A M B D

• Robust and fault-tolerant system• Serves a wide range of workloads and use cases• linearly scalable• Layered Architecture Batch Layer Query Layer Speed Layer

- Nathan Marz

L A M B D

BATCH L AY ER

QU

ERY

LAYE

R

S P E E D / S AV E L AY E R

M A P - S I D E J O I N S( T B ) SCOR I N G

M ATCH I N G S Y S T E M

M ES S AG EBROK ER

BATC H S TORAG E

S P EE D S TORAG E

MER

GE

DATA STORE EVALUATION

C R UDOP ERAT I ON S

THR O UGH PUT40 + M I LL IO N

W R IT ES30+ B I LL I ON

R ECO RDSEA SY T O

M A I NTA I N

CONS IS TE NCY PART IT IO NTO LERA N CE

AVA I L AB I L I T Y

HBASE AS BATCH STORE

T H R O UG H P UT40+ M I LL I O N

W R I T E SC ON S IS T E N CY

AVA I L AB LE

PA RT I T I ONT OL E RA N C E

KAFKA AS BROKER

C ON S IS T E N CY

PA RT I T I ONT O LE RA N C E

AVA I L AB LE

REDIS AS SPEED STORAGE

LO W L AT E N CY

C R U DOP E RAT I ON S

E AS Y TOM A I NTA I N

AS SQL LAYER

QU ER IESI N DE X I N G

T RAN S ACT I O N SM ULT I T E N AN CY

C R U DOP E RAT I ON SE AS Y TO

M A I NTA I N

PHO LIBRARY

QU ER IES

F I LT E R I N G

PHO LIBRARYCONFIGURATION

ANNOTATE THE ENTITY BEAN

@Entity(value="user_matches")public class MatchDataFeedItemDto implements Serializable {

@Embedded private MatchCommunicationElement communication;@Embedded private MatchElement match;

@Property(value = "UID") private long storeUserIdKey; @Property(value = "MID") private long matchId;}

REGISTER THE BEAN

<util:list id="entityPropertiesMappings"> <value>com.eharmony.datastore.model.MatchDataFeedItemDto</value>

</util:list><bean id="entityPropertiesMappingContext" class="com.eharmony.datastore.mapper.EntityPropertiesMappingContext">

<constructor-arg ref="entityPropertiesMappings"/></bean><bean id="entityPropertiesResolver" class="com.eharmony.datastore.mapper.EntityPropertiesResolver">

<constructor-arg ref="entityPropertiesMappingContext"/></bean><bean id="phoenixHBaseQueryTranslator" class="com.eharmony.datastore.hbase.translator.PhoenixHBaseQueryTranslator">

<constructor-arg name="propertyResolver" ref="entityPropertiesResolver" /></bean><bean id="phoenixHBaseQueryExecutor" class="com.eharmony.datastore.hbase.query.executor.PhoenixHBaseQueryExecutor">

<constructor-arg name="queryTranslator" ref="phoenixHBaseQueryTranslator"/> <constructor-arg name="resultMapper" ref="phoenixProjectedResultMapper" />

</bean>

PHO LIBRARYQUERY BUILDING

Disjunction disjunction = new Disjunction(); for (int statusFilter : statusFilters) { disjunction.add(Restrictions.eq("status", statusFilter)); } QueryBuilder.builderFor(FeedItemDto.class).select() .add(Restrictions.eq("userId", userId)) .add(Restrictions.gte("spotlightEnd", spotlightEndDate)) .add(disjunction) .setReturnFields(projection) .addOrder(orderings) .setMaxResults(maxResults) .build();

http://eharmony.github.io/

L A M B D

KA F KA

M A P - S I D E J O I N S( T B ) SCOR I N G BATCH L AY ER

QU E RY L AY E R

S P E E D / S AV E L AY E R

M ATCH I N G S Y S T E M

P E R F O R M A N C E

HBA SE CU TOVE R

S AV E M ATC H R ES P ON S E T I M ES

50% 10 0%

G E T M ATC H E S R E S PON SE T I ME S

HBA SE CUTOV ER 10 0%

SO M E IN S IGH T

C H A L L E N G E S

H O T R E G I O N M I G RAT I O N

metrics.codahale.com

M O N I T O R I N G

http://www.eharmony.com/about/careers/

T H A N K YO UQ U E S T I O N S ?

@ v i j a y v a ng a p a ndu