BETTING THE COMPANY(AGAIN?!) ON A

GRAPH DATABASETHE STORY CONTINUED…

Aseem KishoreOct 2014

http://player.vimeo.com/video/105656434?autoplay=1

MIX GRAPHSBRINGING IDEAS TOGETHER

MIX NEO4J

MIX GRAPHS

STREAMSMATCH (me:User {id: {id}})MATCH (me) <-[:creator]- (creation)

RETURN creation

PAGINATION (BAD)MATCH (me:User {id: {id}})MATCH (me) <-[:creator]- (creation)

RETURN creationORDER BY creation.createdAt DESCSKIP {count} * {page - 1}LIMIT {count}

PAGINATION (GOOD)MATCH (me:User {id: {id}})MATCH (me) <-[:creator]- (creation)

WHERE creation.createdAt < {cursorTime}RETURN creationORDER BY creation.createdAt DESCLIMIT {count}

REMIX FAMILIESMATCH (c:Creation {id: {id}})MATCH (c) -[:remix_source*0..]- (relative)

WHERE relative.createdAt < {cursorTime}RETURN relativeORDER BY relative.createdAt DESCLIMIT {count}

HOME STREAM 1MATCH (me:User {id: {id}})MATCH (me) -[:follows]-> (f) <-[:creator]- (creation)

WHERE creation.createdAt < {cursorTime}RETURN creationORDER BY creation.createdAt DESCLIMIT {count}

HOME STREAM 2MATCH (me:User {id: {id}})MATCH (me) -[:follows]-> (f) -[star:starred]-> (creation)

WITH creation, starORDER BY star.createdAtWITH creation, HEAD(COLLECT(star)) AS star

WHERE star.createdAt < {cursorTime}RETURN creation, star.createdAt AS _starredAtORDER BY _starredAt DESCLIMIT {count}

HOME STREAM 3MATCH (me:User {id: {id}})MATCH (me) -[:starred]-> (c) <-[:remix_source*]- (remix)

WHERE remix.createdAt < {cursorTime}RETURN DISTINCT remixORDER BY remix.createdAt DESCLIMIT {count}

UNION?

UNTIL THEN…nodes = _(results).chain().flatten() .sortBy (node) -> node._orderedAt .unique (node) -> node.id .reverse().value()

Post-processing on our server.

DEDUPING (VERY BAD)MATCH (me:User {id: {id}})MATCH (me) -[:follows]-> (f) -[star:starred]-> (creation)

WITH me, creation, starORDER BY star.createdAtWITH me, creation, HEAD(COLLECT(star)) AS star

MATCH (creation) -[:creator]-> (creator)WHERE NOT (me) -[:follows*0..1]-> (creator)

WHERE star.createdAt < {cursorTime}RETURN creation, star.createdAt AS _starredAtORDER BY _starredAt DESCLIMIT {count}

DEDUPING (BAD)MATCH (me:User {id: {id}})MATCH (me) -[:follows]-> (f) -[star:starred]-> (creation)

WITH me, creation, starORDER BY star.createdAtWITH me, creation, HEAD(COLLECT(star)) AS star

MATCH (creation) -[:creator]-> (creator)WHERE creator <> me AND NOT((me) -[:follows]-> (creator))

WHERE star.createdAt < {cursorTime}RETURN creation, star.createdAt AS _starredAtORDER BY _starredAt DESCLIMIT {count}

QUERY PROFILINGfor key, query of queries echo "Query '#{key}':"

# warm-up: neo4j.query query, params, _

times = [] for i in [1..3] start = Date.now() neo4j.query query, params, _ times.push Date.now() - start

# ... echo "Min/median/max: #{min}/#{median}/#{max} ms. Mean: #{Math.round mean} ms."

(Hat-tip Mark Needham)

HOME STREAM0-following-ids 27 ms1-following-shares 581 ms2-following-features 77 ms3-following-stars 1386 ms4-stars-remixes 189 ms5-shares-remixes 81 ms

All in parallel 1961 ms

(On my aging MacBook Air, for our ~worst-case user.)

IN PRODUCTION…

(But still some || mystery to unravel…)

THRESHOLD

THRESHOLDMATCH (me:User {id: {id}})

WITH me, TOFLOAT(CASE WHEN me.numFollowing < 1 THEN 1 ELSE me.numFollowing END) AS ̀me.numFollowing̀WITH me, FLOOR(LOG(3 * ̀me.numFollowing̀ / 100) / LOG(3)) AS thresholdWITH me, (CASE WHEN threshold < 0 THEN 0 ELSE TOINT(threshold) END) + 1 AS threshold

MATCH (me) -[:follows]-> (following) -[star:starred]-> (creation)

WITH creation, star, thresholdORDER BY star.createdAtWITH creation, COLLECT(star) AS stars, thresholdWHERE LENGTH(stars) >= thresholdWITH creation, stars[threshold - 1] AS star

WITH creation, star.createdAt AS _starredAtORDER BY _starredAt DESCLIMIT {count}

MATCH (creation) -[:creator]-> (creator)RETURN creation, creator, _starredAt

THANK YOU