meetup: an introduction to infinitegraph, and connecting the dots in big data
DESCRIPTION
On August 17, 2011, the InfiniteGraph team hosted a local Meetup attended by dozens of senior developers working on large scale enterprise and startup projects. Big Data problems are quickly presenting themselves in almost every area of computing from Social Network Analysis to File Processing. Many technologies, such as those in the NoSQL space were developed in response to the limitations of current storage systems as an effective mechanism to deal with these mountains of data. And much of that data is interconnected in ways that, when organized properly, gives interesting and often valuable information. InfiniteGraph was designed specifically to traverse complex relationships in big data, and provide the framework for products built to provide real-time network analysis, business decision support and relationship analytics. Speakers: Thomas Krafft, Director of Marketing, InfiniteGraph. Darren Wood, Chief Architect, InfiniteGraph. Mark Maagdenberg, Senior Field Engineer, InfiniteGraph.TRANSCRIPT
Graph Database Overviewand Feature Update
Darren WoodChief Architect, InfiniteGraph
History
• Objectivity – Massively scalable, distributed object oriented database– Used in Government (DoD, Intelligence)
• Machine generated data such as sensor, acoustic…
– OEM Markets • Either complex data models, or high ingest or both
• Significant technical advantage in highly connected (many-to-many) data models
Copyright © InfiniteGraph
Graph Databases
• Key technical attributes• How Infinite Graph addresses these• Query and navigation• Challenges/Requirements of Distribution• Practical applications
Copyright © InfiniteGraph
Graph Databases
• Optimized around data relationships– Relationships as first class citizens– Super fast traversal between entities– Rich/flexible annotation of connections
• Small focused API (typically not SQL)– Natively work with concepts of Vertex/Edge– SQL has no concept of “navigation”– Most attempts based in SQL are convoluted
Copyright © InfiniteGraph
Distributed Graph Must Haves
• High performance distributed persistence• Ability to deal with remote data reads (fast)• Intelligent local cache of subgraphs• Distributed navigation processing• Distributed, multi-source concurrent ingest• Write modes supporting both strict and
eventual consistency
Copyright © InfiniteGraph
Some Code
Copyright © InfiniteGraph
Vertex alice = myGraph.addVertex(new Person(“Alice”)); Vertex bob = myGraph.addVertex(new Person(“Bob”)); Vertex carlos = myGraph.addVertex(new Person(“Carlos”)); Vertex charlie = myGraph.addVertex(new Person(“Charlie”));
alice.addEdge(new Meeting(“Denver”, “5-27-10”), bob);bob.addEdge(new Call(timestamp), carlos);carlos.addEdge(new Payment(100000.00), charlie);bob.addEdge(new Call(timestamp), charlie);
Alice Carlos CharlieBobMeets Calls Pays
Calls
Physical Storage Comparison
Copyright © InfiniteGraph
Meetings
P1 Place TimeP2Alice Denver 5-27-10Bob
Calls
From Time DurationToBob 13:20 25CarlosBob 17:10 15Charlie
Payments
From Date AmountToCarlos 5-12-10 100000Charlie
Met5-27-10Alice
Called13:20Bob
Payed100000Carlos
Charlie
Called17:10
Rows/Columns/Tables Relationship/Graph Optimized
Query and Navigation• Queries – but not as you know them• More like a rules based search and discovery• Asynchronous Results
Copyright © InfiniteGraph
Alice Carlos CharlieBobMeets Calls Pays
Calls
“Find all paths between Alice and Charlie”
“Find all paths between Alice and Charlie – within 2 degrees”
“Find all paths between Alice and Charlie – events in May 2010”
Navigation Example
Copyright © InfiniteGraph
// Create a qualifier that describes the target vertexQualifier findCharliePredicate =
new VertexPredicate(personType, "name == ’Charlie'");
// Construct a navigator which starts with Alice and uses a result qualifier// to find all paths in the graph to CharlieNavigator charlieFinder = alice.navigate(
Guide.SIMPLE_BREADTH_FIRST, // default guide Qualifier.ANY, // no path constraints
findCharliePredicate , // find paths ending with Charlie
myResultHandler); // fire results to supplied handler
// Start the navigatorcharlieFinder.start();
Management of Large Data Graphs
• Graphs grow quickly– Billions of phone calls / day in US– Emails, social media events, IP Traffic– Financial transactions
• Some analytics require navigation of large sections of the graph
• Each step (often) depends on the last• Must distribute data and go parallel
Copyright © InfiniteGraph
Basic Architecture
Copyright © InfiniteGraph
IG Core/API
ConfigurationNavigation Execution
Management Extensions
BlueprintsUser Apps
Objectivity/DB Distributed Database
Session / TX ManagementPlacement
Feature Update
Copyright © InfiniteGraph
2.0
Accelerated Ingest
Copyright © InfiniteGraph
IG Core/API
ConfigurationNavigation Execution
Management Extensions
Session / TX ManagementPlacement
Standard Blocking Ingest/Placement (MDP Plugin)
Objectivity/DB
App-1(Ingest V1)
App-2(Ingest V2)
App-3(Ingest V3)
V1V1 V2
V2 V3V3
App-1(E1 2{ V1V2})
App-2(E23{ V2V3})
App-3
E12E12 E23
E23
Accelerated Ingest
Copyright © InfiniteGraph
IG Core/API
ConfigurationNavigation Execution
Management Extensions
Session / TX Management
Placement(Standard)Placement
(Accelerated)
V1V1
V2V2
V3V3
E12E12
E23E23
Distributed
Pipelines
Sta
ging
Con
tain
ers P
ipeline Containers
E(1->2)
E(3->1)
E(2->3)
E(2->1)
E(2->3)E(3->1)
E(1->2)
E(3->2)
E(1->2)
E(2->3)
E(3->1)
E(2->1)
E(2->3)
E(3->1)
E(3->2)
E(1->2)
InfiniteGraph Visualizer
• Really nice flexible graph viewer• Browser style navigation and history• Full index support – search your data• Display connections around a selected point• Fully customize display to your data model • Full data view via selection
Copyright © InfiniteGraph
InfiniteGraph Visualizer
Copyright © InfiniteGraph
InfiniteGraph Visualizer
Copyright © InfiniteGraph
Indexing Framework
• Focused on providing choice !• Manual Indexes for grouping data• Automatic Indexes for cross population• Query interface with qualification language• Pluggable query operators• External index support (Lucene)
Copyright © InfiniteGraph
• Automated Distributed Navigation• Stored Loadable Navigators• Visualizer Navigation Plugins• More Visualizer Enhancements• More Import/Export support
Copyright © InfiniteGraph
>> next
