what linq is about
DESCRIPTION
Linq & MongoDBTRANSCRIPT
What LINQ is aboutor, Playing with LINQ, MongoDB, and some old new patterns for developers.or, How to inject functional paradigms in imperative languagesor, parallelism made easy.
WebWorkersCamp, 03/07/2010
Pierre Couzy – Microsoft France
Who are you ?
• Worked in the Microsoft space for 20 years– Speaker, trainer, developer, architect, …
• Primarily interested in Web topics– Identity, architecture, WS-*, interop, cloud, …
• Recent interop/OSS activity– Worked on drupal for Sql server project– Works on automated drupal testing on Windows– Speaker at major PHP events
• DrupalCon, PHP Forum, Symfony live
blog.couzy.com / [email protected]
Agenda
• LINQ• More LINQ• LINQ for MongoDB• Going Parallel• Going Distributed• Q&A
• Disclosure: This session is heavily Microsoft-Biased (which is to be expected, I work there)
LINQ : Why ?
• C# is an imperative language• Which is nice• Except for a few things– Splitting data manipulation between your code
and your data store– Coding data manipulation– Applying new manipulation patterns to existing
data
Clumsy c# example
What we’d like is
• Express more intent, less implementation• Easy composition• Strong typing– This is subject to hot debate
• Be as declarative as possible within the limits imposed by C#
From C# to LINQ
Things to notice
• Lazy evaluation– If we step into this code, we see LINQ triggers its magic on
demand• Easy composition
– If we add new clauses after the LINQ expression is built but before it’s consumed, we simply merge the additions to the existing expression
• So far, we’ve worked on objects– Which is how many naive LINQ implementations work :
synchronously pipelining methods calls on collections• LINQ has an interesting twist
– Expression trees
Expression Trees and IQueryable
• Let’s take the same example and introduce AsQueryable()
• Upon execution, we can notice the compiler didn’t eat our LINQ expression– It’s still in there, using a tree format (an
AbstractSymbolTree)• That tree will be resolved at runtime– In the Linq to Objects case, we get exactly the same
behaviour. – The iteration pattern may still be yield-based, but this is left
to IQueryable’s implementation details.
Playing with expression trees
• Let’s tackle Linq To SQL– My expression tree is now pushed to the database
layer– If I want to block the flow to SQL, I just have to
insert a new node in the tree.– Which allows me to balance things• Using C# patterns and methods when I want them• Pushing to the database what it does best
IQueryable goodies
• So, if I want my things to be LINQ aware, I can either– Build some methods and let the IEnumerable<T>
do its pure object magic (I still get lazy eval)– Or, play with IQueryable • And do my own interpretation of the tree
– Let’s see MongoDB in LINQ
• Strongly-typed interaction when querying and updating collections.
• Improved interface to send common Mongo commands (creating indices, getting all the existing dbs and collections, etc.).
• Ultra-fast de/serialization of BSON to .Net CLR types and back.
• LINQ-to-Mongo• Support for Mono
LINQ Patterns
• Having a LINQ Expression is nice: you basically delegate the execution to someone else’s code
• Plus, your LINQ Expressions have every good aspect you usually expect from functional programming (ie, they are closures).
• So, they are perfectly suited to parallelisation• All you have to do is add another node to the
expression tree• Demo : AsParallel()
Writing MapReduce in LINQ public static IQueryable<R> MapReduce<S,M,K,R>(this IQueryable<S> source, Expression<Func<S,IEnumerable<M>>> mapper, Expression<Func<M,K>> keySelector, Expression<Func<K,IEnumerable<M>,R>> reducer){ return source.SelectMany(mapper).GroupBy(keySelector, reducer);}
• Notice that the calling syntax : Source.MapReduce(lMapper, lKeySelector, lReducer)is simply a new node that can be inserted in a larger expression tree.
• This MapReduce method can run on N cores using PLINQ, or on X machines (and N cores per machine) using DryadLINQ
Collection<T> collection;bool IsLegal(Key k);string Hash(Key);
var results = from c in collection where IsLegal(c.key) select new { Hash(c.key), c.value};
DryadLINQ = LINQ + Dryad
C#
collection
results
C# C# C#
Vertexcode
Queryplan(Dryad job)Data
Parallel executionvar logentries = from line in logs where !line.StartsWith("#") select new LogEntry(line);var user = from access in logentries where access.user.EndsWith(@"\ulfar") select access;var accesses = from access in user group access by access.page into pages select new UserPageCount("ulfar", pages.Key, pages.Count());var htmAccesses = from access in accesses where access.page.EndsWith(".htm") orderby access.count descending select access;
Thanks.