linq, take two realizing the linq to everything dream bart j.f. de smet senior software development...
TRANSCRIPT
LINQ, Take TwoRealizing the LINQ to Everything Dream
Bart J.F. De SmetSenior Software Development EngineerMicrosoft Corporation
A Historical Perspective
Censored
7 years ago
Little recent innovation
Where’s the cloud?
Why?
What?How?
Language Integrated QueryMonads
IEnumerable<T>
IQueryable<T>
new[]{ 42 }
SelectMany
IEnumerable<R> SelectMany<T, R>( this IEnumerable<T> source, Func<T, IEnumerable<R>> selector)
Could there be more?
Also see www.codeplex.com/LINQSQO for “Project MinLINQ”
The Monadic Bind Operator Revealed
from _ in name from s in _.ToUpper() select s
name.SelectMany( _ => _.ToUpper(), s => s)Compiler
Null-propagating dot
string s = name?.ToUpper();
Syntactic sugar
One single library function suffices
Building the Maybe MonadFor Fun and No (?) Profit
demo
Bart J.F. De SmetSenior Software Development EngineerCloud Programmability Team
Essential LINQ
var src = new Source<Product>();
var res = from p in src where p.Price > 100m group p by p.Category;
foreach (var p in res) Console.WriteLine(p);
ImplementsIQueryable<T
>
Compiles fine
Does it really have to be a runtime check?
Query Providers RevisitedDo We Need IQueryable<T>?
var res = from p in src
where p.Price > 100m
group p by p.Category;
var res = src
.Where(p => p.Price > 100m)
.GroupBy(p => p.Category);
Syntactic sugar
Can be instance methods
Source<T> Filtered<T> Projected<T>Where Select
No GroupBy “edge”
Leveraging The Query Pattern
var res = from tweet in twitter where tweet.
AboutFromLocation
== “Bart”From
From
AboutLocation
== “LINQ”About
where tweet. About
Query “learns”
class Twitter{ public TwitterByFrom Where(Func<TweetAboutFromLoc, FilterFrom> filter);
// Other filter methods}
Recipe
Method overloadsType state machineOperator overloads
“Has a” type
From operator overloadingclass TwitterByFrom
{ public TwitterByAboutFrom Where(Func<TweetAboutLoc, FilterAbout> filter);
// Other filter methods // Fields with current filters}
select tweet;
Custom syntax trees
Taking It One Step Further
class TweetAboutFromLoc{ public FromString From; public AboutString About; public LocString Location;}
class FromString{ FilterFrom operator ==( FromString f, string s)}
class TweetAboutLoc{ public AboutString About; public LocString Location;}
class AboutString{ FilterAbout operator ==( AboutString f, string s)}
demo
Bart J.F. De SmetSenior Software Development EngineerCloud Programmability Team
Query Providers Revisited
Bottom-Up Tree RewritingA rule drive query provider
+
1*
2 3
Source domain
Rule Set
Patterns with rewrite actions
Target domain
BURS Tree Rewriting Engine
E.g. LINQ Expression Trees E.g. T-SQL Expression Trees
Const(2) Const(3)
Mul
Inc
Currently built at runtime, e.g. once per LINQ provider
that gets loaded
x1 $
Const(x)
State TablesLeaf node
? : int 1
Final states
1 1 $
Any source leaf node with an int constant value can be
reduced using rule 1.
Final states have a cost (incremental, see further).
x
+
a b
1 $Const(x)
2 $Add(a,b)
State TablesLeaf node
? : int 1
Wildcard
a : int 2 *
+ 2 *
2 * 3
Final states
1 1 $
3 2 $
Wildcard b doesn’t get its own entry, since it has the same type as wildcard a.
This reduces space in the BURS tables.
2D table for state transition of the +
operator.
Wildcard matches any subtree that’s assignable to int.
x
+
a b
*
a b
1 $Const(x)
2 $Add(a,b)
3 $Mul(a,b)
State TablesLeaf node
? : int 1
Wildcard
a : int 2 *
+ 2 *
2 * 3
* 2 *
2 * 4
Final states
1 1 $
3 2 $
4 3 $
What’s the meaning of costs?
The cost of 3$ is incremental and has to be added to the cost of all the wildcard matches.
In this case, a match of the Mul-rule (4 ) costs 3$ + costa + costb.
x
+
a b
*
a b
+
c*
a b
1 $Const(x)
2 $Add(a,b)
3 $Mul(a,b)
4 $AddMul(a,b,c)
State TablesLeaf node
? : int 1
Wildcard
a : int 2 *
+ 2 *
2 * 3
4 5
* 2 *
2 * 4
Final states
1 1 $
3 2 $
4 3 $
5 4 $
A pattern forest is kept to map
subtrees on already existent states. Here (a * b) matches state 4.
x
+
a b
*
a b
+
c*
a b
+
a 1
1 $Const(x)
2 $Add(a,b)
3 $Mul(a,b)
4 $AddMul(a,b,c)
1 $Inc(a)
State TablesLeaf node
? : int 1
1 6
Wildcard
a : int 2 *
+ 2 * 6
2 * 3 7
4 5
* 2 *
2 * 4
Final states
1 1 $
3 2 $
4 3 $
5 4 $
7 1 $
Subject Tree Coverings
2 3
* 1
+
Leaf node
? : int 1
1 6
Wildcard
a : int 2 *
+ 2 * 6
2 * 3 7
4 5
* 2 *
2 * 4
Final states
1 1 $
3 2 $
4 3 $
5 4 $
7 1 $
1 $ 1 $
5 $
6 $Constant leaf node 1
doesn’t contribute cost (part of pattern).
demo
Bart J.F. De SmetSenior Software Development EngineerCloud Programmability Team
Bottom Up Rewriting
Environment
MoveN
ext
Got next?
Application
On
Next
Have next!
IEnumerable<T>IEnumerator<T>
IObservable<T>IObserver<T>
Inte
racti
ve R
eactiv
eData Retrieval – Push or Pull?
Event Streams
Towards a unified programming modelProducers are observable sequences
.NET events, WinRT events, sensor APIs, APM methods, tasks, etc.
Consumers are observersHooking up “continuations” or handlers
Observable
Subscribe
Observer
Event Stream Processing Interfacesnamespace System{ public interface IObservable<out T> { IDisposable Subscribe(IObserver<T> observer); }
public interface IObserver<in T> { void OnNext(T value); void OnError(Exception error); void OnCompleted(); }}
Fix
ed
(MS
IL)
Tran
sla
tab
le
(Exp
ressio
n
trees)
ToQueryable
ToObservable
ToEnumerable
AsQ
uery
ab
le
AsEn
um
era
ble
AsQ
bserv
ab
le
AsO
bserv
ab
le
Pull(interactive)
Push(reactive)
Concurre
ncy
(ISch
eduler)
LINQ to *.*
What?
Where
?
How
?
Worker pools Message loopsThreads Distributed
Duality
Hom
o-
icon
ic
IQbservable<T>LINQ to WMI Events
ToQbservableIQueryable<T>
LINQ to SQL
IEnumerable<T>LINQ to Objects
IObservable<T>LINQ to Events
IQbservable<T> - The Dual of IQueryable<T>public interface IQbservable<out T> : IObservable<T>{ Expression Expression { get; } Type ElementType { get; } IQbservableProvider Provider { get; }}
public interface IQbservableProvider{ IQbservable<R> CreateQuery<R>(Expression expression);}
Extended role for some operators
No Execute method
Homoiconic naming
IQbservableIObservable
demo
Bart J.F. De SmetSenior Software Development EngineerCloud Programmability Team
LINQ to Twitter
Baked in notion of “where”?
foreach (var p in res.RemoteOn(new SqlScheduler(“server”))) // Process product
Decoupled “what” from “where”
Entry-point for the schema
Custom schedulers could be very rich (e.g. server farm)
The IScheduler abstractionWhere things happen
var ctx = new NorthwindDataContext(); var res = from product in ctx.Products where product.Price > 100m select product.Name;
Expression Tree Remoting
Rx .NET
RxJSstocks.Where(function (t) { return t.Symbol == “MSFT”; }).Select(function (t) { return t.Quote; })
JSON serializer
Observable data source
Retargeting to AJAX
from ticker in stockswhere ticker.Symbol == “MSFT”select ticker.Quote
Query Server
Alice F9F10
chat
talk
var chat = svc.GetStream<Message>(“chat”);
var talk = from msg in chat where msg.From == “Bob” select Translate(msg.Text, “fr”);
talk.Subscribe(s => ShowMessage(s)); F5
chatchatchat
Where
Select
=
msg “Bob”
From
msg
Text
Translate
Translate
chat
Where
Select
=
msg “Bob”
From
msg
Text
Query Server
Alice
chat
talk
var chat = svc.GetStream<Message>(“chat”);
var talk = from msg in chat where msg.From == “Bob” select Translate(msg.Text, “fr”);
talk.Subscribe(s => ShowMessage(s));
Bob F9
chat
F10
var chat = svc.GetStream(“chat”);
Rx.Observable.FromDOMEvent(txt, “Changed”) .Select(function (e) { return txt.Text; }) .Throttle(.5 /* 500 ms */) .Subscribe(function (s) { chat.OnNext(new Message(“Bob”, s)); }); F5
chat
Where
Select
=
msg “Bob”
From
msg
Text
Translate
chat
chat
Query Server
Alice
chat
talk
Bob
chat
Hello Alice! F5
var chat = svc.GetStream(“chat”);
Rx.Observable.FromDOMEvent(txt, “Changed”) .Select(function (e) { return txt.Text; }) .Throttle(.5 /* 500 ms */) .Subscribe(function (s) { chat.OnNext(new Message(“Bob”, s)); });
Bob says:
Hello Alice!
Hello Alice!
Bonjour Alice!
var chat = svc.GetStream<Message>(“chat”);
var talk = from msg in chat where msg.From == “Bob” select Translate(msg.Text, “fr”);
talk.Subscribe(s => ShowMessage(s)); F5
Bob says:
Bonjour Alice!
Where
Select
=
msg “Bob”
From
msg
Text
Translate
“Bonjour Alice!”
From: “Bob”Text: “Hello Alice!”
From: “Bob”Text: “Hello Alice!”
Translate“Bonjour Alice!”
“Hello Alice!”
demo
Bart J.F. De SmetSenior Software Development EngineerCloud Programmability Team
Expression Tree Serialization
announcing
Now with support for hosting expression tree based queries (enumerable and observable)
StreamInsight v2.1
LINQ to the UnexpectedModel[ Decisions[Reals, SA, VZ], Goals[ Minimize[20 * SA + 15 * VZ] ], Constraints[ C1 -> 0.3 * SA + 0.4 * VZ >= 2000, C2 -> 0.4 * SA + 0.2 * VZ >= 1500, C3 -> 0.2 * SA + 0.3 * VZ >= 500, C4 -> SA <= 9000, C5 -> VZ <= 6000, C6 -> SA >= 0, C7 -> VZ >= 0 ]]
from m in ctx.CreateModel(new { vz = default(double), sa = default(double)})where 0.3 * m.sa + 0.4 * m.vz >= 2000 && 0.4 * m.sa + 0.2 * m.vz >= 1500 && 0.2 * m.sa + 0.3 * m.vz >= 500where 0 <= m.sa && m.sa <= 9000 && 0 <= m.vz && m.vz <= 6000orderby 20 * m.sa + 15 * m.vzselect m
To compute call Solve
Symbolic, non-persisted
Cost / barrel
Max# barrels
Min# barrels
Refin
em
en
t %
LINQ to the Unexpected
from costSA in GetPriceMonitor("SA")from costVZ in GetPriceMonitor("VZ")
from m in ctx.CreateModel(new { vz = default(double), sa = default(double) })
where 0.3 * m.sa + 0.4 * m.vz >= 2000 && 0.4 * m.sa + 0.2 * m.vz >= 1500 && 0.2 * m.sa + 0.3 * m.vz >= 500where 0 <= m.sa && m.sa <= 9000 && 0 <= m.vz && m.vz <= 6000
orderby costSA * m.sa + costVZ * m.vzselect m
Observabledata sources
Sele
ctM
an
y
Parametersto decision
Subscribe here!
demo
Bart J.F. De SmetSenior Software Development EngineerCloud Programmability Team
LINQ to Constraints
Related Content
DEV413 – Curing your Event Processing Blues using Reactive Extensions (Rx)
Find Me Later The Week At The Ask The Experts
Resources
Connect. Share. Discuss.
http://europe.msteched.com
Learning
Microsoft Certification & Training Resources
www.microsoft.com/learning
TechNet
Resources for IT Professionals
http://microsoft.com/technet
Resources for Developers
http://microsoft.com/msdn
Evaluations
http://europe.msteched.com/sessions
Submit your evals online
© 2012 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to
be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS
PRESENTATION.