graph store

Linked data analytics in an ad-system (Slide outlines)

Inder Singh, Srikanth Sundarrajan@inmobi

User store

• Why a store for user data– Just like advertiser or publisher in the network,

consumer/user is a very important entity• It is an entity that has associated activity

• Not an attribute on a network entity

• Value for network, advertiser & publisher by showing ads that are very relevant to users– We need to understand users better

– And leverage information about them better

What do we need to store?

• Activities that a user is involved in

• Profile of the user (ex. Demographics)

• Besides• Location

• Device

• Apps etc

User Data Model

User: (Identifier, Age, Gender,

Interest, Preference, …)

User Data Model

User: (Identifier, Age, Gender,

Interest, Preference, …)

Site: Platform, Category …

Visits

User Data Model

User: (Identifier, Age, Gender,

Interest, Preference, …)

Site: Platform, Category …

Visits

Visit: Time of day, Requests, Impressions,

Clicks, Downloads, Burn, Engagement Time…

User Data Model

User: (Identifier, Age, Gender,

Interest, Preference, …)

Site: Platform, Category …

Visits

Served: Impressions, Clicks, Downloads, Burn,

Engagement Time…

AdGrp: Category, Objective …

Served

User Data Model

Owns: Request, Impressions, Clicks, Downloads, Burn.…

User: (Identifier, Age, Gender,

Interest, Preference, …)

Site: Platform, Category …

Visits

AdGrp: Category, Objective …

Served

Geo:

loca

ted

Device: manufacturer, OS

type, …

owns

How does the data look at scale ?

U1

S1 S2

U2

U3

U4

U5

S3

Ad1

Ad2

C1

C2

D1

D2

What can we do with this data?

• Examples– Get an user’s detail and user’s network activity to

infer something about the user

– Target segment of users based on user’s attributes or aggregate activity

– Understand reach of a targeting criteria

• Further– Function of how efficiently can we store this data

and how quickly can we retrieve information

About User Data

• Too sparse & High Cardinality (> billion)

• Random id (Quality of data)

• Some popular network entities are associated with large number of users (> 100 million)

• Lot of attributes about Users are inferred and as we have stronger signals, these need to be mutated

How ?

• Store for Analytics / Insights– Is all about organizing data aligning with retrieval

use cases

– Retrieval time = Organization / Data Size

– What is Organization • Simply put: Trade space for time

Popular storage structures

• Rows & Columns – Relational Table– Indexing ?

• Each column ?

• Group of columns ?

• Ingestion cost

• Cardinality

– Ideal if the queries/data extraction is reasonably well defined and conforms to set patterns

– Appends is the most efficient way of ingestion

Popular storage structures

• Columnar storage (big table / db)– Key based lookup / scan

– Optimized for use cases where not all data stored for the key needs to be retrieved

– Mutations / Append patterns are both scalable patterns for ingestion

Heavily Indexed Relation

An optimized representation

Dimension User Bitmap

Site1 + Adgroup1 1000011001……

Site1 + SFO 0001111000….

Site2 + nexus 1000011110…

….. …..

….. ….

…. ….

….. ….

Starting point….

• Can do all kinds of set operations to get you user reach

• Found a user for Site1+Adgroup1 combination. Find other apps/devices this user came from during a time interval. – Walking the graph if I had a link from userid to Dimension?

Dimension User Bitmap

Site1 + Adgroup1 1000011001……

Site1 + SFO 0001111000….

Site2 + nexus 1000011110…

….. …..

….. ….

…. ….

….. ….

Starting point extended (Logical Diagram) ….

UserID

u1

u2

u3

u4

• Now we can walk from a userID we found in bitmap back to where all other places it occurs….life is cool ☺

Engineers, we want neat abstractions so life is uncool again..

Directed multi property graphs

Buzzwords in Graph world

• Neo4j

• Titan

• Dex

• Pregel

• Giraph

• Gremlin

• Tinker Pop blueprint

What to do?

• Evaluate leading graph db’s neo4j, titan, orientDB, flockdb(twitter), Facebook TAO.

• Titan & neo4j – Challenges we faced

• SuperNodes• Queries like give me Sites, Devices with exclusive users

or in general class of queries requiring lots of edge traversals over lots of super nodes never returned for hours and the DB server goes in GC.

• Talk to experts from neo4j, titan. Still not there for huge scale and expensive queries.

• Research paper of DEX

• Formalizes and provides abstractions over our thinking of using bitmaps

• Compares class of queries it supports against leading graph DB’s and results are very promising.

Graph: Formal definition

• V = {v1,...,vn} : Finite Set of vertices

• E = {e1, ...,em} : Finite Set of Edges

• Relation Sets– T = {(e1,t1),...,(em,tm)} is the set of tail pairs (ei , ti ),

which indicates that the tail of ei is the vertex ti ∈ V , – H = {(e1,h1),...,(em,hm)} is the set of head pairs (ei , hi

), which indicates that the head of ei is the vertex hi ∈ V .

Formal definition contd..• Given an object o, which is either an edge or

vertex (o ∈ {V ∪ E}), we map a single label to each object L = {(o,l) | o ∈ (V ∪ E),l ∈ string}

• Attributes : Ai = {(o1,c1),...,(or,cr)}, which assign an attribute value ci ∈ D (where D are the valid data types such as int, boolean, timestamp, etc.)

• G = (V,E,L,T,H,A1,...,Ap)

Some things we want

• Store large #objects sets and access efficiently

• Given a key, find matching objects (Vertex/Edges)

• Given and object retrieve set of values associated with this object.

Assumptions

• Vertex/Edges in the entire graph have a unique ID called oid (object identifier).

So what’s the real deal

• Value Sets : Group all objects matching a value together. Similar to inverted index.

• Two set of maps : – VALUE_OID : maps a value to a bitmap

– OID_VALUE : maps an oid to a value

LABELS

• Two set of maps

• Value_oid : alike inverted index

• OID_VALUE_MAP : allows to walkLinks in graph

TAILS

• Value_oid : invertex index of all edges outgoing from this vertex

• Oid_value : given an edgeID what’s the outgoing vertex

HEADS

• Value_oid : inverted indexOf all incoming edges on a vertex

• Oid_value : go from oid to it’s value

Attributes

• Table name = Attr_KEY_NAME

• Value_oid : inverted index of all edge/Vertex for this value

• Oid_value : find value for this attributeKey given oid

Efficient bitmaps

• Parition 64 bitmap space into significantly big ranges for each entitytype like device, site, etc.

• Results in less sparse bitmaps and better compression.

Primitive apis : objects

• Bitmap Objects(NameSpace n, Value v) : works on VALUE_OID map

– Example • Objects(Age, 28) : Table = “Age”, Value = “28” return

bitmap of Vertex(User) with age = “28”

VALUE BITMAP(OIDS)

28 100111000….

17 011000………

OID

456

789

645

Objects()

Primitive apis : lookup

• V lookup(Namespace n, Long oid) : works on oid_value table.

• Example – Lookup(Age, 456) returns value = 17

VALUE BITMAP(OIDS)

28 100111000….

17 011000………

OID

456

789

645

Primitive apis : domain

• Iterable<Key, Value> Domain(Namespace n)

• Example – Domains(Age) returns an iterator over the value_oid table keys

VALUE BITMAP(OIDS)

28 100111000….

17 011000………

OID

456

789

645

Primitive apis : insert, remove

• Insert(Namespace n, Long oid, Value v)

• remove(Namespace n, Long oid, Value v)

Tinker Pop apis

• Let’s look at the code of AbstractGraph, ShadowfaxEdge, ShadowFaxvertex to understand how all of this works together

Walking the graph

• Find all users who own iphone and have 100 clicks in system. Find common sites among these users.

• Let’s break it down– Users who own iphone = objects(“label”,

“iphone”) – returns a bitmap with one vertex set of iphone.

Walking the graph

• From iphone vertex let’s go to all OWNS edges– Bitmap1 Objects(HEADS, “iphone-vid”) : bitmap

for all edges incident into iphone vertex

– Bitmap2 Objects(LABELS, “OWNS”) all owns edges

– all OWNS edges incident into iphone-vid = Bitmap3 = (Bitmap1 AND bitmap2)

Walking the graph contd..

• Find all edges which have “clicks = 100” – Bitmap3 = objects(click, “100”)

• Bitmap4 = (Bitmap3 AND bitmap4) All edges incident into iphone vertex and have 100 clicks.

Walking the graph contd..

• For all OWNS edges incident into iphone vertex let’s walk to users.– For (Long oid : Bitmap.vector()) { //find vertex from edge

Long userVertexOID = lookup(TAILS, oid); //find Sites visited by this user i.e. walk from user to site : getEdges out of this vertex which have LABEL “visits” Lookup(HEADS, (objects(TAILS, userVertexOID) AND objects(LABELS, “VISITS”) ));

}

Is that the way to write code?

• No use tinker pop api’s we give, life is easy ☺• How our implementation gives more power –

Expensive queries optimized through native apis.

Ingestion at Scale

Volatile Graph

Volatile graph

Volatile Graph

Volatile Graph

In memory Graphs

Local Graph

Local Graph

Local Graph

Local Graph

Persistent Graphs

Merge MergeMerge Merge

Global Graph

Merge local persistent graphs

Shard1

u1 s1

s2

d1

u10

Shard2

u11 s1

s2

d1

u20

Parition on UserID and replicate metadata to all Shards

Build up

• KV store : high throughput, low latency for bigger sized values.– Evaluated

• LevelDB : Chosen at the moment

• LigthingDB

• Others

Build up contd..

• Bitmap Indexing Library– Evaluated

• Fastbit : Chosen

• Javaewah

Examples of costly queries

• getEntitiesWithMaxUserCount(startDate, endDate, entityType) – Ex : work on a batch of sites in parallel. For a site for multiple dates work in parallel.

• getEntitiesWithExclusiveUsers(startDate, endDate, entityType)

• getRepeatingUserDistribution(startDate, endDate, entityType)