FS2for

Fun & Profit

@adilakhter

1

The next 45 minutes

... is about FS2

2

@ load.ivy("co.fs2" %% "fs2-core" % "0.9.0-M3")

@ load.ivy("co.fs2" %% "fs2-io" % "0.9.0-M3")

3

4

5

fs2.util.

Task

6

Taskis a

Monadic context for Asynchronous computation

7

Task[A]

8

Taskseparates

what to do from when to do

9

@ def launchMissile() = ... // side-effectlaunchMissile: ()Unit

10

@ val launchFuture = Future { launchMissile() }launchFuture: scala.concurrent.Future[Unit] = ...// A missile has already launched!

11

@ launchFuture// nothing happens due to side-effect memoization!

12

@ val launchTask = Task { launchMissile() }launchTask: fs2.util.Task[Unit]= ...

13

@ launchTask.unsafeRun // <- a missile now launched

14

@ launchTask.unsafeRun // missile 1 -> launched@ launchTask.unsafeRun // missile 2 -> launched

15

Taskseparates

what to do from when to do

16

Taskis

purely functional & compositional

17

Task Combinators

4 now[A](a: A): Task[A]

4 delay[A](a: => A): Task[A]

4 fail(e: Throwable): Task[Nothing]

4 fork[A](a: => Task[A]): Task[A]

18

Example: Retrieve Tweets

@ import twitter4j._

@ def statuses(query: Query): Task[List[Status]] = Task { twitterClient.search(query).getTweets.toList}

19

FS2

20

Goals

4 Incremental IO & Stream Processing

4 Modularity & Composability

4 Resource-safety

4 Performance

21

Goals

4 Incremental IO & Stream Processing

4 Modularity & Composability

4 Resource-safety

4 Performance

22

@ import fs2._@ import fs2.util._

@ implicit val strategy: Strategy = Strategy.fromFixedDaemonPool(8)@ implicit val scheduler: Scheduler = Scheduler.fromFixedDaemonPool(8)

23

File IO using FS2import java.nio.file.Paths

def fahrenheitToCelsius(f: Double): Double = (f - 32.0) * (5.0/9.0)

val streamConverter: Task[Unit] = io.file.readAll[Task](Paths.get("testdata/fahrenheit.txt"), 4096) .through(text.utf8Decode) .through(text.lines) .filter(s => !s.trim.isEmpty && !s.startsWith("//")) .map(line => fahrenheitToCelsius(line.toDouble).toString) .intersperse("\n") .through(text.utf8Encode) .through(io.file.writeAll(Paths.get("testdata/celsius.txt"))) .run

streamConverter.unsafeRun()

24

FS2

... provides an abstraction to declaratively specify how to obtain stream of data.

25

Stream[F[_], O]

26

27

28

29

Example: Pure Streams

@ Stream(1,2,3)res29: Stream[Nothing, Int] = Segment(Emit(Chunk(1, 2, 3)))

@ Stream(1,2,3) ++ Stream(4,5,6)res30: Stream[Nothing, Int] = append(Segment(Emit(Chunk(1, 2, 3))), Segment(Emit(Chunk(()))).flatMap(<function1>))

30

Stream ≡ Emit | Await | Done

31

Emit

Emit[F[_],O]( head: Seq[O], tail: Stream[F, O] = Done)

32

Example: Pure Streams

@ import Stream._import Stream._

@ emit(1) // emits valueres30: Stream[Nothing, Int] = Segment(Emit(Chunk(1)))

@ emits(Seq(1,4,10,20)) // <- emits sequenceres31: Stream[Nothing, Int] = Segment(Emit(Chunk(1, 4, 10, 20)))

33

Await

Await[F[_], I, O]( req: F[I], // side-effect recv: I ⇒ Stream[F, O]) // continuation of Stream.

34

Example: Effectful Stream

@ def statuses(query: Query): Task[List[Status]] = ...

35

// Builds a Twitter Search Process for a given Query@ def tweetStream(query: Query): Stream[Task, Status] = { val queryTask: Task[List[Status]] = statuses(query) eval(queryTask).flatMap { statusList ⇒ emitAll(statusList) }}

36

//In essence, it builds: Stream: Await ⇒ (_ ⇒ Emit ⇒ Done)

val searchStream = tweetStream(new Query("#spark"))

37

Example: A Stream of Positive Integers

@ import Stream._import Stream._

@ def integerStream: Stream[Task, Int] = { def next (i: Int) : Stream[Task, Int] = eval(Task(i)).flatMap { i ⇒ emit(i) ++ next(i + 1) } next(1) }// in essence, it describes Stream of {1, 2, 3, 4 ....}

38

Stream[F, O]

... represents a stream of O valueswhich can interleave external requeststo evaluate computation of form F[_]

39

Stream Combinators

4 Stream.constant

4 Stream.eval

4 Stream.repeatEval

4 Stream.evalMap

4 fs2.io.readAll

4 ...40

Evaluating a Stream

41

Stream.run

@ integerStream.take(10)res23: Stream[Task, Int] = ...

@ integerStream.take(10).runres24: Task[Unit] = Task

@ integerStream.take(10).run.unsafeRun

42

Stream.runLog

scala> integerStream.take(10).runLog.unsafeRunres4: Vector[Int] = Vector(1, 2, 3, 4, 5, 6, 7, 8, 9, 10)

43

Stream.runLog

What does happen when we execute integerStream.runLog.run?@ def integerStream: Stream[Task, Int] = { def next (i: Int) : Stream[Task, Int] = eval(Task(i)).flatMap { i ⇒ emit(i) ++ next(i + 1) } next(1) }

@ integerStream.runLog.unsafeRun // <--- ?

44

Other Run Combinators

> def runFree: fs2.util.Free[F,Unit]

> def runFold[B](z: B)(f: (B, A) => B): F[B]

45

Transformations

46

Stream.map

@ val stream = integerStream.map(_ * 100)

stream: Stream[Task, Int] =attemptEval(Task).flatMap(<function1>).flatMap(<function1>).mapChunks(<function1>)

@ stream.take(5).runLog.unsafeRunres39: Vector[Int] = Vector(100, 200, 300, 400, 500)

47

Stream.flatMap

@ integerStream.flatMap { i => emits(List.fill(i)(i)) } .take(10) .runLog .unsafeRun

res45: Vector[Int] = Vector(1, 2, 2, 3, 3, 3, 4, 4, 4, 4)

48

Stream.zip

@ val zippedStream = integerStream zip emitAll(Seq("A", "B"))res52: Stream[Task, (Int, String)] =evalScope(Scope(Bind(Eval(Snapshot),<function1>))).flatMap(<function1>)

@ zippedStream.runLog.unsafeRunres54: Vector[(Int, String)] = Vector((1, "A"), (2, "B"))

49

Composability

50

Pipe

> Stream[F, A].through(Pipe[A, B]) => Stream[F, B]

51

Pipe

type Pipe[F[_],-I,+O] = Stream[F,I] => Stream[F,O]

52

Pipe

@ import pipe._import pipe._

53

Example

@ integerStream |> filter( _%2 == 0)res59: Stream[Task, Int] = ...

54

Example

@ integerStream |> filter( _%2 == 0) |> exists(_ == 10)res60: Stream[Task, Boolean] = ...

55

Pipe2

type Pipe2[F[_],-I,-I2,+O] = (Stream[F,I], Stream[F,I2]) => Stream[F,O]

56

Sink

> Stream[F, A].to(Sink[F, A]) => Stream[F, Unit]

57

Sink

type Sink[F[_],-I] = Pipe[F,I,Unit]

58

Example

@ def log[A]: Sink[Task, A] = _.evalMap{ x => Task{ println(s"$x") } }

59

Example

scala> val streamWithSink: Stream[Task, Unit] | = integerStream.take(5) to printIntsscala> streamWithSink.run.unsafeRun12345

60

ExampleSentiment Analysis of Tweets

61

62

Building Source

@ def src: Stream[Task, Query] = awakeEvery[Task](15 seconds).map { _ ⇒ new Query("#spark") }

63

Twitter Query Pipe

// defined earlierdef statuses(query: Query): Task[List[Status]] = ...

def twitterPipe: Pipe[Task, Query, List[Status]] = _.evalMap{ query ⇒ statuses(query) }

64

Stream Pipeline

src .through(twitterPipe) // Stream[Task, List[Status]] .flatMap(Stream.emits(_)) // Stream[Task, Status] .map(s ⇒ Tweet(s)) // Stream[Task, Tweet]

65

Sentiment Analysis Pipe

@ import SentimentAnalyzer._

@ def analyze(t: Tweet): Task[EnrichedTweet] = Task { EnrichedTweet( t.author, t.body, t.retweetCount, sentiment(t.body)) }

66

@ def analysisPipe: Pipe[Task, Tweet, EnrichedTweet] = _.evalMap{tweet ⇒ analyze(tweet)}

67

Stream Pipeline (so far)

src .through(twitterPipe) // Stream[Task, List[Status]] .flatMap(Stream.emits(_)) // Stream[Task, Status] .map(s ⇒ Tweet(s)) // Stream[Task, Tweet]

68

Stream Pipeline

src .through(twitterPipe) // Stream[Task, List[Status]] .flatMap(Stream.emits(_)) // Stream[Task, Status] .map(s ⇒ Tweet(s)) // Stream[Task, Tweet]

.through(analysisPipe) // Stream[Task, EnrichedTweet]

69

Stream Pipeline

@ val tweetStream = src .through(twitterPipe) // Stream[Task, List[Status]] .flatMap(Stream.emits(_)) // Stream[Task, Status] .map(s ⇒ Tweet(s)) // Stream[Task, Tweet]

.through(analysisPipe) // Stream[Task, EnrichedTweet]

70

Evaluating Stream Pipeline

@ val consoleTweetStream = tweetStream.to(snk) // Stream[Task, Unit]

@ consoleTweetStream.run.unsafeRun

71

72

Connecting Stream with WS

// http4s Websocket Routecase r @ GET -> Root / "websocket" ⇒ val query = new Query("#spark") val stream = tweetStream(query).map(Text(_))

WS(Exchange(stream, ...)) // <- connected stream to WS

73

74

Merge & Join

75

@ val s = integerStream .through(randomDelays(1.second)) .through(log("s"))

@ val t = range(1, 3) .through(randomDelays(1.second)) .through(log("t"))

76

Interleave

@ s interleave t res89: Stream[Task, Int] = ...

@ (s interleave t).through(log("interleaved")).runLog.unsafeRun

s > 1t > 1interleaved > 1interleaved > 1s > 2t > 2interleaved > 2interleaved > 2s > 3res90: Vector[Int] = Vector(1, 1, 2, 2)

77

Merge@ (s merge t).through(log("merged")).runLog.unsafeRuns > 1merged > 1t > 1merged > 1s > 2merged > 2s > 3merged > 3t > 2merged > 2s > 4merged > 4s > 5merged > 5....

78

Either

@ s either tres22: Stream[Task, Either[Int, Int]] = ...

@ Stream.emit("A, B, C") either tres23: Stream[Task, Either[String, Int]]

79

Join

@ val u = Stream.range(10, 20)u: Stream[Nothing, Int] = ...@@ val streams: Stream[Task, Stream[Task, Int]] = Stream(s, t, u)streams: Stream[Task, Stream[Task, Int]] = ...

@ concurrent.join(3)(streams)res27: Stream[Task, Int] = ...

80

“There are two types oflibraries: the ones people hateand the ones nobody use"

— Unknown

81

Thank You

82

CreditsCredits for the images used in slides: Mario Sixtus.

References and Further Readings4 FS2: The Official Guide

4 Scalaz-Stream Masterclass by Runar at NE Scala 2016

4 Scalaz Task - the missing documentation

4 scalaz-stream User Notes

4 Comparing akka-stream and scalaz-stream with code examples

4 Additional Resources

83