Download - Logging in Scala
![Page 1: Logging in Scala](https://reader031.vdocuments.site/reader031/viewer/2022021801/5888ab781a28ab80248b4ea1/html5/thumbnails/1.jpg)
Logging in Scala
John Nestor 47 Degrees
www.47deg.com
August 9, 2016
147deg.com
![Page 2: Logging in Scala](https://reader031.vdocuments.site/reader031/viewer/2022021801/5888ab781a28ab80248b4ea1/html5/thumbnails/2.jpg)
47deg.com © Copyright 2016 47 Degrees
Outline
• Introduction
• Architecture
• Features
• Implementation
• Demos
• Cross Service Log Aggregation
2
![Page 3: Logging in Scala](https://reader031.vdocuments.site/reader031/viewer/2022021801/5888ab781a28ab80248b4ea1/html5/thumbnails/3.jpg)
Introduction
3
![Page 4: Logging in Scala](https://reader031.vdocuments.site/reader031/viewer/2022021801/5888ab781a28ab80248b4ea1/html5/thumbnails/4.jpg)
47deg.com © Copyright 2016 47 Degrees
Existing Scala Loggers
• Java loggers (used in Scala code)
• Log4j (used in Spark)
• SLF4J and Logback
• Scala loggers
• Abandoned: Configgy, Logula (Coda Hale)
• Log4s (wraps SLF4J)
• scala-logging (Heiko Seeberger) (wraps Log4j)
• Special Scala loggers
• Akka actor logging (can route to Logback)
• Play logging (uses Logback)
4
![Page 5: Logging in Scala](https://reader031.vdocuments.site/reader031/viewer/2022021801/5888ab781a28ab80248b4ea1/html5/thumbnails/5.jpg)
47deg.com © Copyright 2016 47 Degrees
Why a New Logger?
• Pure Scala. (No Java code)
• Richer messages (Json rather than text strings)
• Aggregate logs across multiple (micro-)services
• Clean design with rich features
5
![Page 6: Logging in Scala](https://reader031.vdocuments.site/reader031/viewer/2022021801/5888ab781a28ab80248b4ea1/html5/thumbnails/6.jpg)
47deg.com © Copyright 2016 47 Degrees
New Scala Logger History
• Whitepages Logger - part of a new Scala web service framework
• Persist Logger (work supported by 47 Degrees)
• clean up and improve API
• removed Whitepages specific stuff
• added complete documentation
• Full source and documentation on Github
• Maven central "com.persist" % "persist-logging_2.11" % "1.2.4"
• Apache 2 license
6
![Page 7: Logging in Scala](https://reader031.vdocuments.site/reader031/viewer/2022021801/5888ab781a28ab80248b4ea1/html5/thumbnails/7.jpg)
47deg.com © Copyright 2016 47 Degrees
Rich Logging Example
log.error(s”Too many values Size:${vals.size} Max:${vals.max}”)
log.error(map(“msg”->”Too many values”, ”Size”->vals.size, ”Max”->vals.max), id=currentId)
{“msg”:”Too Many Values”, “Size”: 10, “Max: “33}
7
![Page 8: Logging in Scala](https://reader031.vdocuments.site/reader031/viewer/2022021801/5888ab781a28ab80248b4ea1/html5/thumbnails/8.jpg)
Architecture
8
![Page 9: Logging in Scala](https://reader031.vdocuments.site/reader031/viewer/2022021801/5888ab781a28ab80248b4ea1/html5/thumbnails/9.jpg)
47deg.com © Copyright 2016 47 Degrees
Persist Logger Architecture
9
logging.conf
logback.xml
Scala Logger API Slf4j API
Akka Actor Logging API
Custom Logback
Appender
LogBack Custom Akka Logging Actor
Logging Actor
Stdout Appender
File Appender
Kafka Appender
Other Appenders
Logger State
Scala Logger
Control API
![Page 10: Logging in Scala](https://reader031.vdocuments.site/reader031/viewer/2022021801/5888ab781a28ab80248b4ea1/html5/thumbnails/10.jpg)
Features
10
![Page 11: Logging in Scala](https://reader031.vdocuments.site/reader031/viewer/2022021801/5888ab781a28ab80248b4ea1/html5/thumbnails/11.jpg)
47deg.com © Copyright 2016 47 Degrees
API Log Call
• Level
• Rich Message
• Id (optional)
• Per request level control
• Per request log aggregation
• Exception (optional)
• use RichException for a RichMessage
11
![Page 12: Logging in Scala](https://reader031.vdocuments.site/reader031/viewer/2022021801/5888ab781a28ab80248b4ea1/html5/thumbnails/12.jpg)
47deg.com © Copyright 2016 47 Degrees
Rich Messages
• String
• Int, Long, Float, Double, BigDecimal
• null, Boolean
• Seq[RichMessage]
• Map[String,RichMessage]
12
![Page 13: Logging in Scala](https://reader031.vdocuments.site/reader031/viewer/2022021801/5888ab781a28ab80248b4ea1/html5/thumbnails/13.jpg)
47deg.com © Copyright 2016 47 Degrees
Why Json?
• Easy to support rich metadata in addition to the message
• Easier to parse in downstream tools
• No need to manually escape strings
• Naturally supports multiline messages
• Standard text format
• But many internal possibilities
• This logger uses Persist Json (only internally)
• Rich messages use standard Scala types for in internal API
13
![Page 14: Logging in Scala](https://reader031.vdocuments.site/reader031/viewer/2022021801/5888ab781a28ab80248b4ea1/html5/thumbnails/14.jpg)
47deg.com © Copyright 2016 47 Degrees
Kind of Logs
• Error log
• Alternative logs
• GC
• Timing (for fine grain timing)
• Server (request-response, duration)
• Client (request, response, duration)
• User defined
14
![Page 15: Logging in Scala](https://reader031.vdocuments.site/reader031/viewer/2022021801/5888ab781a28ab80248b4ea1/html5/thumbnails/15.jpg)
47deg.com © Copyright 2016 47 Degrees
Standard Fields
• @category (error, gc, …)
• @host
• @service
• @severity (info, debug, warn, error, …)
• @timestamp (msec)
• class, file, line, actor
• msg
• trace (for exceptions, Json)
15
![Page 16: Logging in Scala](https://reader031.vdocuments.site/reader031/viewer/2022021801/5888ab781a28ab80248b4ea1/html5/thumbnails/16.jpg)
47deg.com © Copyright 2016 47 Degrees
Appenders
• Stdout
• no server/service specific fields
• color (optional)
• summary
• File (daily rotation)
• Kafka
• Custom, user defined
• Control destination
• Control contents and format
16
![Page 17: Logging in Scala](https://reader031.vdocuments.site/reader031/viewer/2022021801/5888ab781a28ab80248b4ea1/html5/thumbnails/17.jpg)
47deg.com © Copyright 2016 47 Degrees
Level Control
• logging.conf (default options), can override
• uses Typesafe config
• levels: API, SLF4J, Akka
• Change level via API
• Per request custom level
• Custom filters
• Based on level, and content
17
![Page 18: Logging in Scala](https://reader031.vdocuments.site/reader031/viewer/2022021801/5888ab781a28ab80248b4ea1/html5/thumbnails/18.jpg)
47deg.com © Copyright 2016 47 Degrees
Timing
• Fine grain timing
• Uses request ids to aggregate
• Time in microseconds
• Time log
• Can be used with concurrency (futures and actors)
18
![Page 19: Logging in Scala](https://reader031.vdocuments.site/reader031/viewer/2022021801/5888ab781a28ab80248b4ea1/html5/thumbnails/19.jpg)
47deg.com © Copyright 2016 47 Degrees
Other Loggers
• Captures messages from other legacy loggers that might be used in new project libraries
• SLF4J (captured via Logback appender)
• Akka Actor Logging (captured by custom Akka log handing actor)
19
![Page 20: Logging in Scala](https://reader031.vdocuments.site/reader031/viewer/2022021801/5888ab781a28ab80248b4ea1/html5/thumbnails/20.jpg)
47deg.com © Copyright 2016 47 Degrees
Look At Documentation
• logging.conf
• Overview
• API
20
![Page 21: Logging in Scala](https://reader031.vdocuments.site/reader031/viewer/2022021801/5888ab781a28ab80248b4ea1/html5/thumbnails/21.jpg)
Implementation
21
![Page 22: Logging in Scala](https://reader031.vdocuments.site/reader031/viewer/2022021801/5888ab781a28ab80248b4ea1/html5/thumbnails/22.jpg)
47deg.com © Copyright 2016 47 Degrees
Persist Logger Architecture
22
logging.conf
logback.xml
Scala Logger API Slf4j API
Akka Actor Logging API
Custom Logback
Appender
LogBack Custom Akka Logging Actor
Logging Actor
Stdout Appender
File Appender
Kafka Appender
Other Appenders
Logger State
Scala Logger
Control API
![Page 23: Logging in Scala](https://reader031.vdocuments.site/reader031/viewer/2022021801/5888ab781a28ab80248b4ea1/html5/thumbnails/23.jpg)
47deg.com © Copyright 2016 47 Degrees
Dealing with Concurrency
• The logging actor provides fully generalized sync
• Serializes output
• Handles filters
• But asking the actor on every message can be slow
• @volatile booleans for each log level associated with logger
• Note the logger itself is global state
• One copy for entire app, no need to pass a parameter everywhere: ClassLogging and ActorLogging traits.
• Must start before any messages and stop after any messages (but there is a development workaround)
• Contains log level booleans23
![Page 24: Logging in Scala](https://reader031.vdocuments.site/reader031/viewer/2022021801/5888ab781a28ab80248b4ea1/html5/thumbnails/24.jpg)
47deg.com © Copyright 2016 47 Degrees
Source Location
• Could find using exception stack trace at run-time
• Better solution is to use Scala macro and reflection at compile-time
24
![Page 25: Logging in Scala](https://reader031.vdocuments.site/reader031/viewer/2022021801/5888ab781a28ab80248b4ea1/html5/thumbnails/25.jpg)
Demos
25
![Page 26: Logging in Scala](https://reader031.vdocuments.site/reader031/viewer/2022021801/5888ab781a28ab80248b4ea1/html5/thumbnails/26.jpg)
47deg.com © Copyright 2016 47 Degrees
Demos
• Simple
• Actor
• Exceptions
• Request Id
• Alternative
• Other (Actor, SLF4J)
• Timing
• Filter
• Appender
26
![Page 27: Logging in Scala](https://reader031.vdocuments.site/reader031/viewer/2022021801/5888ab781a28ab80248b4ea1/html5/thumbnails/27.jpg)
Cross Service Log Aggregation
27
![Page 28: Logging in Scala](https://reader031.vdocuments.site/reader031/viewer/2022021801/5888ab781a28ab80248b4ea1/html5/thumbnails/28.jpg)
47deg.com © Copyright 2016 47 Degrees
The Problem
• Applications are being split into ever more micro-services
• Much harder to answer questions?
• What is the overall control flow for various requests?
• What was the root cause of an error?
• Where are the performance bottlenecks?
• In this talk we look at a prototype system that can be used to answer these kinds of questions
• Work in progress
28
![Page 29: Logging in Scala](https://reader031.vdocuments.site/reader031/viewer/2022021801/5888ab781a28ab80248b4ea1/html5/thumbnails/29.jpg)
47deg.com © Copyright 2016 47 Degrees
Related Work
• Google Dapper
• Zipkin
• Akka Tracing (wraps Zipkin)
29
![Page 30: Logging in Scala](https://reader031.vdocuments.site/reader031/viewer/2022021801/5888ab781a28ab80248b4ea1/html5/thumbnails/30.jpg)
47deg.com © Copyright 2016 47 Degrees
Approach
• Use Scala as much as possible
• Aggregate logs for each independent request across a set of Scala services
• Provide a basis for both
• Near real-time
• Alerts
• View aggregated logs for any request
• More comprehensive batch analysis
• Failure statistics
• Overall patterns
• Trends over time
30
![Page 31: Logging in Scala](https://reader031.vdocuments.site/reader031/viewer/2022021801/5888ab781a28ab80248b4ea1/html5/thumbnails/31.jpg)
47deg.com © Copyright 2016 47 Degrees
Overall Architecture
31
KafkaSpark
Streaming(Aggregation)
NoSql DB
REST Log API *
(Merge, Filter)
Services *
* Uses: Persist Service Framework
Persist Service Framework Uses:Akka HTTPPersist Json
Persist Logger
Spark Batch Analytics
![Page 32: Logging in Scala](https://reader031.vdocuments.site/reader031/viewer/2022021801/5888ab781a28ab80248b4ea1/html5/thumbnails/32.jpg)
47deg.com © Copyright 2016 47 Degrees
Scala REST Service
32
Server Log
Service(can be lots of these)
Error Log
Client Log
... To Other Services
Kafka(only one cluster/topic for all
logs and services)
... From Other Services
![Page 34: Logging in Scala](https://reader031.vdocuments.site/reader031/viewer/2022021801/5888ab781a28ab80248b4ea1/html5/thumbnails/34.jpg)
47deg.com © Copyright 2016 47 Degrees
Aggregation (via Spark Streaming)
• Pass from client to server
• Client name
• Tracking Id
• same for a single request across all micro services
• Span Id
• unique for each call for a given Client-Server pair
• Match Tracking Id and Span Id across
• Client log
• Server log
34
![Page 35: Logging in Scala](https://reader031.vdocuments.site/reader031/viewer/2022021801/5888ab781a28ab80248b4ea1/html5/thumbnails/35.jpg)
47deg.com © Copyright 2016 47 Degrees
Cross Service Matching
35
Id1Span1
X
AId1
Span1X
Id1Span2
A
Id1Span3
A
AId1
Span1Span2
B
BId1
Span2A
BId1
Span3A
AId1
Span1Span3
B
Match
Match
Match
Match
Service A Service B
![Page 36: Logging in Scala](https://reader031.vdocuments.site/reader031/viewer/2022021801/5888ab781a28ab80248b4ea1/html5/thumbnails/36.jpg)
47deg.com © Copyright 2016 47 Degrees
Look at Sample Output
• Array of log messages
• Tree of log messages
• Filtered tree with timings
36
![Page 37: Logging in Scala](https://reader031.vdocuments.site/reader031/viewer/2022021801/5888ab781a28ab80248b4ea1/html5/thumbnails/37.jpg)
Questions
37