windy city db - recommendation engine with neo4j
TRANSCRIPT
![Page 1: Windy City DB - Recommendation Engine with Neo4j](https://reader036.vdocuments.site/reader036/viewer/2022062405/554f65aab4c905bb178b4a13/html5/thumbnails/1.jpg)
Adding a Recommendation
EngineWindyCityDBMax De Marzi
![Page 2: Windy City DB - Recommendation Engine with Neo4j](https://reader036.vdocuments.site/reader036/viewer/2022062405/554f65aab4c905bb178b4a13/html5/thumbnails/2.jpg)
About Me
• My Blog: http://maxdemarzi.com• Find me on Twitter: @maxdemarzi• Email me: [email protected]• GitHub: http://github.com/maxdemarzi
Built the Neography Gem (Ruby Wrapper to the Neo4j REST API)Playing with Neo4j since 10/2009
![Page 3: Windy City DB - Recommendation Engine with Neo4j](https://reader036.vdocuments.site/reader036/viewer/2022062405/554f65aab4c905bb178b4a13/html5/thumbnails/3.jpg)
Agenda
• What is Neo4j?• What is Neography?• Approaches• Gremlin• Gremlin Recommends• Cypher• Cypher Recommends
![Page 4: Windy City DB - Recommendation Engine with Neo4j](https://reader036.vdocuments.site/reader036/viewer/2022062405/554f65aab4c905bb178b4a13/html5/thumbnails/4.jpg)
What is Neo4j?
• A Graph Database + Lucene Index• Property Graph• Full ACID (atomicity, consistency, isolation,
durability)• High Availability (with Enterprise Edition)• 32 Billion Nodes, 32 Billion Relationships,
64 Billion Properties• Embedded Server (Java or JVM languages)• REST API (for everyone else)
![Page 5: Windy City DB - Recommendation Engine with Neo4j](https://reader036.vdocuments.site/reader036/viewer/2022062405/554f65aab4c905bb178b4a13/html5/thumbnails/5.jpg)
Obligatory CAP theorem Slide
• Neo4j is at the section joining Consistency with Availability just like your RDBMS
![Page 6: Windy City DB - Recommendation Engine with Neo4j](https://reader036.vdocuments.site/reader036/viewer/2022062405/554f65aab4c905bb178b4a13/html5/thumbnails/6.jpg)
Good For
• Highly connected data (social networks)• Recommendations (e-commerce)• Path Finding (how do I know you?)
• A* (Least Cost path)• Data First Schema (bottom-up, but you still
need to design)
![Page 7: Windy City DB - Recommendation Engine with Neo4j](https://reader036.vdocuments.site/reader036/viewer/2022062405/554f65aab4c905bb178b4a13/html5/thumbnails/7.jpg)
Property Graph
![Page 8: Windy City DB - Recommendation Engine with Neo4j](https://reader036.vdocuments.site/reader036/viewer/2022062405/554f65aab4c905bb178b4a13/html5/thumbnails/8.jpg)
If you’ve ever
• Joined more than 7 tables together• Modeled a graph in a table• Written a recursive CTE• Tried to write some crazy stored procedure
with multiple recursive self and inner joins
You should use Neo4j
![Page 9: Windy City DB - Recommendation Engine with Neo4j](https://reader036.vdocuments.site/reader036/viewer/2022062405/554f65aab4c905bb178b4a13/html5/thumbnails/9.jpg)
What is Neography?
![Page 10: Windy City DB - Recommendation Engine with Neo4j](https://reader036.vdocuments.site/reader036/viewer/2022062405/554f65aab4c905bb178b4a13/html5/thumbnails/10.jpg)
A very thin wrapper to the Neo4j REST API
• Two layers:Following the REST APIRuby Sugar
• Made for graph database programmer happiness >8-]
• Read the github Readme and Specs• Can I haz pull requests?• Want Active Record?
Use the activerecord-neo4j-adapter gem which sits on top of neography
![Page 11: Windy City DB - Recommendation Engine with Neo4j](https://reader036.vdocuments.site/reader036/viewer/2022062405/554f65aab4c905bb178b4a13/html5/thumbnails/11.jpg)
How do I install Neo4j?
Create a project directory and do this now if you haven’t yet. Make sure wget is installed.
![Page 12: Windy City DB - Recommendation Engine with Neo4j](https://reader036.vdocuments.site/reader036/viewer/2022062405/554f65aab4c905bb178b4a13/html5/thumbnails/12.jpg)
Approaches
![Page 13: Windy City DB - Recommendation Engine with Neo4j](https://reader036.vdocuments.site/reader036/viewer/2022062405/554f65aab4c905bb178b4a13/html5/thumbnails/13.jpg)
Collaborative Filtering
• Step 1: Collect User Behavior
• Step 2: Find similar Users
• Step 3: Recommend Behavior taken by similar users
![Page 14: Windy City DB - Recommendation Engine with Neo4j](https://reader036.vdocuments.site/reader036/viewer/2022062405/554f65aab4c905bb178b4a13/html5/thumbnails/14.jpg)
Content Based Filtering
• Step 1: Collect Item Characteristics
• Step 2: Find similar Items
• Step 3: Recommend Similar Items
Marko likes Romantic Zombie Comedies, what other romantic zombie comedies are there?
Tweet him @twarko
![Page 15: Windy City DB - Recommendation Engine with Neo4j](https://reader036.vdocuments.site/reader036/viewer/2022062405/554f65aab4c905bb178b4a13/html5/thumbnails/15.jpg)
Hybrid
• Combine the two for better results.
• Example: Netflix
![Page 16: Windy City DB - Recommendation Engine with Neo4j](https://reader036.vdocuments.site/reader036/viewer/2022062405/554f65aab4c905bb178b4a13/html5/thumbnails/16.jpg)
What is Gremlin?
![Page 17: Windy City DB - Recommendation Engine with Neo4j](https://reader036.vdocuments.site/reader036/viewer/2022062405/554f65aab4c905bb178b4a13/html5/thumbnails/17.jpg)
Gremlin is
• A Graph Traversal Language• A domain specific language for traversing
property graphs• Implemented by most Graph Database
Vendors• Primarily seen with the Groovy Language• With JVM connectivity in Java, Scala, and
other languages
![Page 18: Windy City DB - Recommendation Engine with Neo4j](https://reader036.vdocuments.site/reader036/viewer/2022062405/554f65aab4c905bb178b4a13/html5/thumbnails/18.jpg)
Marko Rodriguezhttp://markorodriguez.com
Created by:
![Page 19: Windy City DB - Recommendation Engine with Neo4j](https://reader036.vdocuments.site/reader036/viewer/2022062405/554f65aab4c905bb178b4a13/html5/thumbnails/19.jpg)
A Graph DSL
A Dynamic Language for the JVM
A Data Flow Framework
“JDBC” for Graph DBs
![Page 20: Windy City DB - Recommendation Engine with Neo4j](https://reader036.vdocuments.site/reader036/viewer/2022062405/554f65aab4c905bb178b4a13/html5/thumbnails/20.jpg)
Gremlin Recommends
![Page 21: Windy City DB - Recommendation Engine with Neo4j](https://reader036.vdocuments.site/reader036/viewer/2022062405/554f65aab4c905bb178b4a13/html5/thumbnails/21.jpg)
Hybrid movie recommendations
![Page 22: Windy City DB - Recommendation Engine with Neo4j](https://reader036.vdocuments.site/reader036/viewer/2022062405/554f65aab4c905bb178b4a13/html5/thumbnails/22.jpg)
Our Graph (from MovieLens)
![Page 23: Windy City DB - Recommendation Engine with Neo4j](https://reader036.vdocuments.site/reader036/viewer/2022062405/554f65aab4c905bb178b4a13/html5/thumbnails/23.jpg)
Recommendation Algorithm
m = [:];x = [] as Set; v = g.v(node_id);
v. out('hasGenre').aggregate(x).back(2).inE('rated').filter{it. stars > 3}.
(continued)outV. outE('rated'). filter{it.stars > 3}.inV.filter{it != v}.filter{it.out('hasGenre').toSet().equals(x)}.groupCount(m){\"${it.id}:${it.title}\"}.iterate();m.sort{a,b -> b.value <=> a.value}[0..24]
![Page 24: Windy City DB - Recommendation Engine with Neo4j](https://reader036.vdocuments.site/reader036/viewer/2022062405/554f65aab4c905bb178b4a13/html5/thumbnails/24.jpg)
Explanation
m = [:];x = [] as Set; v = g.v(node_id);
In Groovy [:] is a map, we will return thisThe set “x” will hold the collection of genres we want our recommendedmovies to have.
v is our starting point.
![Page 25: Windy City DB - Recommendation Engine with Neo4j](https://reader036.vdocuments.site/reader036/viewer/2022062405/554f65aab4c905bb178b4a13/html5/thumbnails/25.jpg)
Explanation
v. out('hasGenre'). (we are now at a genre node)aggregate(x).
We fill the empty set “x” with the genres of our movie.These are the properties we want to make sure our recommendations have.
![Page 26: Windy City DB - Recommendation Engine with Neo4j](https://reader036.vdocuments.site/reader036/viewer/2022062405/554f65aab4c905bb178b4a13/html5/thumbnails/26.jpg)
Explanation
back(2). (we are back to our starting point)inE('rated').filter{it. stars > 3}. (we are now at the link between our movie and users)
We go back two steps to our starting movie, go to the relationship ‘rated’ and filter it so we only keep those with more than 3 stars.
![Page 27: Windy City DB - Recommendation Engine with Neo4j](https://reader036.vdocuments.site/reader036/viewer/2022062405/554f65aab4c905bb178b4a13/html5/thumbnails/27.jpg)
Explanation
outV. (we are now at a user node)outE('rated'). filter{it.stars > 3}. (we are now at the link between user and movie)
We follow our relationships to the users who made them, and then go to the “rated” relationships of movies which also received morethan 3 stars.
![Page 28: Windy City DB - Recommendation Engine with Neo4j](https://reader036.vdocuments.site/reader036/viewer/2022062405/554f65aab4c905bb178b4a13/html5/thumbnails/28.jpg)
Explanation
inV. (we are now at a movie node) filter{it != v}.
We follow our relationships to the movies who received the, but filter out “v” which is our starting movie. We do not want the system to recommend the same movie we just watched.
![Page 29: Windy City DB - Recommendation Engine with Neo4j](https://reader036.vdocuments.site/reader036/viewer/2022062405/554f65aab4c905bb178b4a13/html5/thumbnails/29.jpg)
Explanation
filter{it.out('hasGenre').toSet().equals(x)}.
We also want to keep only the movies that have the same genres as ourstarting movie. People gave Toy Story and Terminator both 4 stars,but you wouldn’t want to recommend one to the other.
![Page 30: Windy City DB - Recommendation Engine with Neo4j](https://reader036.vdocuments.site/reader036/viewer/2022062405/554f65aab4c905bb178b4a13/html5/thumbnails/30.jpg)
Explanation
groupCount(m){\"${it.id}:${it.title}\"}.iterate();
groupCount does what it sounds like and stores the values in the map “m”we created earlier, but we to retain the id and title of the movies.
iterate() is needed from the Neo4j REST API, the gremlin shell does it automatically for you. You will forget this one day and kill30 minutes of your life trying to figure out why you get nothing.
![Page 31: Windy City DB - Recommendation Engine with Neo4j](https://reader036.vdocuments.site/reader036/viewer/2022062405/554f65aab4c905bb178b4a13/html5/thumbnails/31.jpg)
Explanation
m.sort{a,b -> b.value <=> a.value}[0..24]
Finally, we sort our map by value in descending order and grab the top25 items… and you’re done.See http://maxdemarzi.com/2012/01/16/neo4j-on-heroku-part-two/for the full walk-through including data loading.
![Page 32: Windy City DB - Recommendation Engine with Neo4j](https://reader036.vdocuments.site/reader036/viewer/2022062405/554f65aab4c905bb178b4a13/html5/thumbnails/32.jpg)
What is Cypher?
![Page 33: Windy City DB - Recommendation Engine with Neo4j](https://reader036.vdocuments.site/reader036/viewer/2022062405/554f65aab4c905bb178b4a13/html5/thumbnails/33.jpg)
The blue pill after the red pill
• Graph Query language for Neo4j• Based on Pattern Matching• Makes querying easy again
![Page 34: Windy City DB - Recommendation Engine with Neo4j](https://reader036.vdocuments.site/reader036/viewer/2022062405/554f65aab4c905bb178b4a13/html5/thumbnails/34.jpg)
ASCII ART FTW: a--b, a-->b, a<--c
Cypher : Neo4j Query Language
![Page 35: Windy City DB - Recommendation Engine with Neo4j](https://reader036.vdocuments.site/reader036/viewer/2022062405/554f65aab4c905bb178b4a13/html5/thumbnails/35.jpg)
Cypher Recommends
![Page 36: Windy City DB - Recommendation Engine with Neo4j](https://reader036.vdocuments.site/reader036/viewer/2022062405/554f65aab4c905bb178b4a13/html5/thumbnails/36.jpg)
Similar UsersWhich users have rated movies that I have rated within one star of my rating?
![Page 37: Windy City DB - Recommendation Engine with Neo4j](https://reader036.vdocuments.site/reader036/viewer/2022062405/554f65aab4c905bb178b4a13/html5/thumbnails/37.jpg)
Average RatingWhat is the average rating(by similar users) of a movie not rated by me?
![Page 38: Windy City DB - Recommendation Engine with Neo4j](https://reader036.vdocuments.site/reader036/viewer/2022062405/554f65aab4c905bb178b4a13/html5/thumbnails/38.jpg)
Movies I should seeWhich movies (that I haven't seen) have been rated 4 stars or higher by users similar to me?
![Page 39: Windy City DB - Recommendation Engine with Neo4j](https://reader036.vdocuments.site/reader036/viewer/2022062405/554f65aab4c905bb178b4a13/html5/thumbnails/39.jpg)
Questions?
?
![Page 40: Windy City DB - Recommendation Engine with Neo4j](https://reader036.vdocuments.site/reader036/viewer/2022062405/554f65aab4c905bb178b4a13/html5/thumbnails/40.jpg)
Thank you!http://maxdemarzi.com