eharmony @ phoenix con 2016
TRANSCRIPT
V I JAY VA N G A PA N D U
@ v i j a y v a ng a p a nd uv i j ay ku marv a ng a p a nd u
W H O A R E W E ?
EH AR M O NY CR EAT E STH E H A PP I ES T ,M O S T PAS S I O NAT EAND M O S T FU L F I L L I NG R EL AT I O NS H I PS *
*ACC OR D I NG TO A R E C E N T ST U DY
4 3 8M A R R I AG E S P E R DAY
150 questions
PersonalityValuesAttributesBeliefs
M ATC H I N G S Y S T E M
Compatibility Matching System®
C OM PAT IB I L I T Y MATC H ING A F F IN I T Y MATC H ING MATC H
D IS T R I B U T I ON
IntellectEnergy
SociabilityAmbition
KindnessCuriosity
HumorSpiritualit
y
U S E R U P DAT E S
M ATC H D E L I V E RY (V 1 )
M A P-S I DE J O I N S( TB ) SC OR I N G
VOL DE M ORT
MATC H DATA S E RV I CE
MATC H I NG S YS TE M
30+ M I LL I ON E VE NT S
65+ M I L L I ON U SE RS30+ B I L L I ON R E C ORD S
VO L D E M O RT ?T H AT N A M E
T H AT N A M ES O U N D S FAM I L I A R
VO L DE M O RT
AUT OPART I T I O N I NG
P LU G GAB L E SE R IAL I ZAT IO N
AU T OR EP L IC AT I O N
KE Y-VA LUE DY NAM O
G O SS I P
U S E R U P DAT E S O ( N ^ 2 )
* USE R UPDAT ED T HE AD DR E SS
N E E D F O R S C A L AB I L I T Y
VOL DE MORT
30+ Million Match Events / Day
30+ Billion Match Records
Millions of user generated Events / Day
Low latency user requests
1 . 4G B / M IN (14 )
N E E D F O R S C A L AB I L I T YG E T M ATC H E S R E S P ON S E T I M E S
DATA STORE NEEDS
QU E R I E S
LOW L AT E N CY
C R U DOP E RAT I ON S F I LT E R I N G
T H R O U G H P UT40+ M I LL I O N
W R I T E S
30+ B I LL I ONR E C OR D S
DATA STORE NEEDS
E AS Y TOM A I NTA I N
C ON S IS T E N CY AVA I L AB LEPA RT I T I ON
T O LE RA N C E
BREAKING CAP ?
Consistency
Availability PartitionTolerance
CA CP
AP
MongoDBHBaseRedis
CassandraDynamoDB
Riak
RDBMSKAFKA
L A M B D
• Robust and fault-tolerant system• Serves a wide range of workloads and use cases• linearly scalable• Layered Architecture Batch Layer Query Layer Speed Layer
- Nathan Marz
L A M B D
BATCH L AY ER
QU
ERY
LAYE
R
S P E E D / S AV E L AY E R
M A P - S I D E J O I N S( T B ) SCOR I N G
M ATCH I N G S Y S T E M
M ES S AG EBROK ER
BATC H S TORAG E
S P EE D S TORAG E
MER
GE
DATA STORE EVALUATION
C R UDOP ERAT I ON S
THR O UGH PUT40 + M I LL IO N
W R IT ES30+ B I LL I ON
R ECO RDSEA SY T O
M A I NTA I N
CONS IS TE NCY PART IT IO NTO LERA N CE
AVA I L AB I L I T Y
HBASE AS BATCH STORE
T H R O UG H P UT40+ M I LL I O N
W R I T E SC ON S IS T E N CY
AVA I L AB LE
PA RT I T I ONT OL E RA N C E
KAFKA AS BROKER
C ON S IS T E N CY
PA RT I T I ONT O LE RA N C E
AVA I L AB LE
REDIS AS SPEED STORAGE
LO W L AT E N CY
C R U DOP E RAT I ON S
E AS Y TOM A I NTA I N
AS SQL LAYER
QU ER IESI N DE X I N G
T RAN S ACT I O N SM ULT I T E N AN CY
C R U DOP E RAT I ON SE AS Y TO
M A I NTA I N
PHO LIBRARY
QU ER IES
F I LT E R I N G
PHO LIBRARYCONFIGURATION
ANNOTATE THE ENTITY BEAN
@Entity(value="user_matches")public class MatchDataFeedItemDto implements Serializable {
@Embedded private MatchCommunicationElement communication;@Embedded private MatchElement match;
@Property(value = "UID") private long storeUserIdKey; @Property(value = "MID") private long matchId;}
REGISTER THE BEAN
<util:list id="entityPropertiesMappings"> <value>com.eharmony.datastore.model.MatchDataFeedItemDto</value>
</util:list><bean id="entityPropertiesMappingContext" class="com.eharmony.datastore.mapper.EntityPropertiesMappingContext">
<constructor-arg ref="entityPropertiesMappings"/></bean><bean id="entityPropertiesResolver" class="com.eharmony.datastore.mapper.EntityPropertiesResolver">
<constructor-arg ref="entityPropertiesMappingContext"/></bean><bean id="phoenixHBaseQueryTranslator" class="com.eharmony.datastore.hbase.translator.PhoenixHBaseQueryTranslator">
<constructor-arg name="propertyResolver" ref="entityPropertiesResolver" /></bean><bean id="phoenixHBaseQueryExecutor" class="com.eharmony.datastore.hbase.query.executor.PhoenixHBaseQueryExecutor">
<constructor-arg name="queryTranslator" ref="phoenixHBaseQueryTranslator"/> <constructor-arg name="resultMapper" ref="phoenixProjectedResultMapper" />
</bean>
PHO LIBRARYQUERY BUILDING
Disjunction disjunction = new Disjunction(); for (int statusFilter : statusFilters) { disjunction.add(Restrictions.eq("status", statusFilter)); } QueryBuilder.builderFor(FeedItemDto.class).select() .add(Restrictions.eq("userId", userId)) .add(Restrictions.gte("spotlightEnd", spotlightEndDate)) .add(disjunction) .setReturnFields(projection) .addOrder(orderings) .setMaxResults(maxResults) .build();
http://eharmony.github.io/
L A M B D
KA F KA
M A P - S I D E J O I N S( T B ) SCOR I N G BATCH L AY ER
QU E RY L AY E R
S P E E D / S AV E L AY E R
M ATCH I N G S Y S T E M
P E R F O R M A N C E
HBA SE CU TOVE R
S AV E M ATC H R ES P ON S E T I M ES
50% 10 0%
G E T M ATC H E S R E S PON SE T I ME S
HBA SE CUTOV ER 10 0%
SO M E IN S IGH T
C H A L L E N G E S
H O T R E G I O N M I G RAT I O N
http://www.eharmony.com/about/careers/
T H A N K YO UQ U E S T I O N S ?
@ v i j a y v a ng a p a ndu