counters and async infrastructures · counters and async infrastructures ... multiple counters...

48

Upload: vutuyen

Post on 26-Jul-2018

271 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Counters and Async Infrastructures · Counters and Async Infrastructures ... Multiple Counters [2016-02-16 15:18:27,626] ... • Using enum semantics instead of “register new counter”
Page 2: Counters and Async Infrastructures · Counters and Async Infrastructures ... Multiple Counters [2016-02-16 15:18:27,626] ... • Using enum semantics instead of “register new counter”

Counters and AsyncInfrastructures

Proposal for Code ContributionGuy Sela, Senior Engineer, HPE

Page 3: Counters and Async Infrastructures · Counters and Async Infrastructures ... Multiple Counters [2016-02-16 15:18:27,626] ... • Using enum semantics instead of “register new counter”

Who am I ?• Senior Engineer at HPE• Served as a developer and team leader at the Israeli NSA• Developing software since 2004

E-mail: [email protected]: http://www.linkedin.com/pub/guy-sela/70/17/9aa

Page 4: Counters and Async Infrastructures · Counters and Async Infrastructures ... Multiple Counters [2016-02-16 15:18:27,626] ... • Using enum semantics instead of “register new counter”

AgendaCounters Infrastructure• Why do we need Counters?• Proposed Implementation

• Capabilities• Examples• Under the Hood

• Third-Party Alternatives

Async Infrastructure• Scope• Overview• Under the Hood

Page 5: Counters and Async Infrastructures · Counters and Async Infrastructures ... Multiple Counters [2016-02-16 15:18:27,626] ... • Using enum semantics instead of “register new counter”

Why do we need Counters?

• High-Frequency events will swamp the Log.• Good alternative for Log.debug() and Log.trace()• Useful information for troubleshooting.• Powerful tool for analyzing bottlenecks.• What events happened in my system until now?• Analytics

Page 6: Counters and Async Infrastructures · Counters and Async Infrastructures ... Multiple Counters [2016-02-16 15:18:27,626] ... • Using enum semantics instead of “register new counter”

Proposed Implementation

• Light-weight infrastructure that is very easy to integrate

into your code

• Supports the common use-cases for counters

• Found to be extremely useful for monitoring and

troubleshooting production systems

Page 7: Counters and Async Infrastructures · Counters and Async Infrastructures ... Multiple Counters [2016-02-16 15:18:27,626] ... • Using enum semantics instead of “register new counter”

Capabilities

• Incremental & State Counters

• Configurable interval for “print counters delta to the log”

• Runtime counters silencing based on the user’s log library

• Almost no overhead on the system

• Counter querying API based on regular expressions

• REST API

Page 8: Counters and Async Infrastructures · Counters and Async Infrastructures ... Multiple Counters [2016-02-16 15:18:27,626] ... • Using enum semantics instead of “register new counter”

Use Case – Incremental Counterpublic class UserService {

// High-Scale Eventvoid userEvent() {

log.info(“Event Called”); // Will swamp the Log… // Business Logic

}}

Page 9: Counters and Async Infrastructures · Counters and Async Infrastructures ... Multiple Counters [2016-02-16 15:18:27,626] ... • Using enum semantics instead of “register new counter”

Use Case – Incremental Counterpublic class UserService {

// High-Scale Eventvoid userEvent() {

UserServiceCounters.user_event_called.inc();… // Business Logic

}}

Page 10: Counters and Async Infrastructures · Counters and Async Infrastructures ... Multiple Counters [2016-02-16 15:18:27,626] ... • Using enum semantics instead of “register new counter”

Log Output Example – 1 Minute Interval[2016-02-16 14:18:27,626] INFO : (com.hpe.test.TestClassA:252) – Did Something[2016-02-16 14:18:27,967] INFO : (com.hpe.test.TestClassB:254) – Did Something Else[2016-02-16 14:18:28,028] CNT : user_event_called : +37824[2016-02-16 14:18:45,125] INFO : (com.hpe.test.TestClassC:63) – Yet Another Log Line[2016-02-16 14:19:28,028] CNT : user_event_called : +41215[2016-02-16 14:19:50,026] INFO : (com.hpe.test.TestClassC:72) – I Love Log Lines[2016-02-16 14:20:30,028] CNT : user_event_called : +51621[2016-02-16 14:20:50,026] INFO : (com.hpe.test.TestClassA:11) – Log Lines Rock!

10

Page 11: Counters and Async Infrastructures · Counters and Async Infrastructures ... Multiple Counters [2016-02-16 15:18:27,626] ... • Using enum semantics instead of “register new counter”

Log Output Example – Multiple Counters[2016-02-16 15:18:27,626] INFO : (com.hpe.test.TestClassA:252) – Did Something[2016-02-16 15:18:27,967] INFO : (com.hpe.test.TestClassB:254) – Did Something Else[2016-02-16 15:18:28,028] CNT : user_event_called : +37824, other_counter: +72[2016-02-16 15:18:45,125] INFO : (com.hpe.test.TestClassC:63) – Yet Another Log Line[2016-02-16 15:19:28,028] CNT : user_event_called : +41215, foo_counter: +17

11

Page 12: Counters and Async Infrastructures · Counters and Async Infrastructures ... Multiple Counters [2016-02-16 15:18:27,626] ... • Using enum semantics instead of “register new counter”

public class UserService {// Boilerplate Code to define a new Counters Groupenum UserServiceCounters {

user_event_called(), other_user_counter(), third_counter();

private OccurenceCounter counter; public void inc() {

counter.inc();}

UserServiceCounters() {counter = new OccurenceCounter(“UserServiceCounters”, name());

}}

}12

import com.hpe.counter.OccurrenceCounter;

Page 13: Counters and Async Infrastructures · Counters and Async Infrastructures ... Multiple Counters [2016-02-16 15:18:27,626] ... • Using enum semantics instead of “register new counter”

Counter Types

• Incremental Counter – Value is manipulated using inc(). The log prints show the delta from the last log print. No delta No log print.

• State Counter – Value is manipulated using set(). The log prints show the current value of the counter in case it changed. Useful for detecting queue build-up.

13

Page 14: Counters and Async Infrastructures · Counters and Async Infrastructures ... Multiple Counters [2016-02-16 15:18:27,626] ... • Using enum semantics instead of “register new counter”

Counters SilencingLog4J Config File // Standard mechanism to change Log Severity

log4j.logger.UserService.user_event_called=WARN

14

Page 15: Counters and Async Infrastructures · Counters and Async Infrastructures ... Multiple Counters [2016-02-16 15:18:27,626] ... • Using enum semantics instead of “register new counter”

Use Case - Using Counters to Identify Bottlenecksvoid myBusinessLogic {

String value = DB.getKey(“key”);// …Socket.sendOnSocket(new Packet(value.toBytes()));// …invokeIntensiveMethod(value);

}Scenario: 5000 Events per second which invoke myBusinessLogic();Symptom:We only manage to handle 2000 Events per second

15

Page 16: Counters and Async Infrastructures · Counters and Async Infrastructures ... Multiple Counters [2016-02-16 15:18:27,626] ... • Using enum semantics instead of “register new counter”

Use Case - Using Counters to Identify Bottlenecksvoid myBusinessLogic {

BLCounters.before_db.inc();String value = DB.getKey(“key”);

BLCounters.after_db.inc();// …BLCounters.before_socket.inc();Socket.sendOnSocket(new Packet(value.toBytes()));BLCounters.after_socket.inc();// …BLCounters.before_intensive.inc();invokeIntensiveMethod(value);BLCounters.after_intensive.inc();

}16

Page 17: Counters and Async Infrastructures · Counters and Async Infrastructures ... Multiple Counters [2016-02-16 15:18:27,626] ... • Using enum semantics instead of “register new counter”

Log Output will tell us where is the Bottleneck[2016-02-16 15:18:28,028] CNT : before_db: +5000,

after_db: +2000, before_socket: +2000, after_socket:

+2000, before_intensive: +2000, after_intensive: +2000

The DB is Slowing Us!

17

Page 18: Counters and Async Infrastructures · Counters and Async Infrastructures ... Multiple Counters [2016-02-16 15:18:27,626] ... • Using enum semantics instead of “register new counter”

Log Output will tell us where is the Bottleneck[2016-02-16 15:18:28,028] CNT : before_db: +5000,

after_db: +5000, before_socket: +5000, after_socket:

+5000, before_intensive: +5000, after_intensive: +2000

The Intensive Method is Slowing Us!

18

Page 19: Counters and Async Infrastructures · Counters and Async Infrastructures ... Multiple Counters [2016-02-16 15:18:27,626] ... • Using enum semantics instead of “register new counter”

What events happened in my System until Now?

CountersDumperService APIString getCounters(String regexp);• Call using REST API / Other Management Console

19

Page 20: Counters and Async Infrastructures · Counters and Async Infrastructures ... Multiple Counters [2016-02-16 15:18:27,626] ... • Using enum semantics instead of “register new counter”

Analytics

20

Page 21: Counters and Async Infrastructures · Counters and Async Infrastructures ... Multiple Counters [2016-02-16 15:18:27,626] ... • Using enum semantics instead of “register new counter”

Under the Hood

• Counters based on AtomicLong & AtomicInteger causes

insignificant overhead.

• Using enum semantics instead of “register new counter”

implementation

• Background Thread that writes the delta of the counters to

the Log in each configurable interval.

21

Page 22: Counters and Async Infrastructures · Counters and Async Infrastructures ... Multiple Counters [2016-02-16 15:18:27,626] ... • Using enum semantics instead of “register new counter”

Third-Party Alternatives

• Known third-party implementations with similar goals:

Metrics, Parfait, JAMon, Java Simon, Perf4J.

• We haven’t made a deep comparison between the

proposed solution to these.

22

Page 23: Counters and Async Infrastructures · Counters and Async Infrastructures ... Multiple Counters [2016-02-16 15:18:27,626] ... • Using enum semantics instead of “register new counter”

Questions?

23

Page 24: Counters and Async Infrastructures · Counters and Async Infrastructures ... Multiple Counters [2016-02-16 15:18:27,626] ... • Using enum semantics instead of “register new counter”

And Now to a Completely Different Subject

24

Page 25: Counters and Async Infrastructures · Counters and Async Infrastructures ... Multiple Counters [2016-02-16 15:18:27,626] ... • Using enum semantics instead of “register new counter”

Async Infrastructure

25

Page 26: Counters and Async Infrastructures · Counters and Async Infrastructures ... Multiple Counters [2016-02-16 15:18:27,626] ... • Using enum semantics instead of “register new counter”

Introduction

• In ODL’s Hydrogen version, every OSGI Bundle was activated using the BundleActivator pattern.

• Every OSGI Service declared its implementations and dependencies in its Activator.

• After the transformation to MD-SAL, Plugins/Applications define their dependencies, implementations and API using YANG Model.

• Applications that reside in the Controller communicate with each other using MD-SAL: RPCs and Notifications.

26

Page 27: Counters and Async Infrastructures · Counters and Async Infrastructures ... Multiple Counters [2016-02-16 15:18:27,626] ... • Using enum semantics instead of “register new counter”

Introduction• Application can be implemented by a number of OSGI Services

and Bundles.• The presented Async Infrastructure facilitates communication

between OSGI services, and is agnostic to YANG, RPCs, etc.• The infrastructure is currently only relevant for Bundles that still

use the BundleActivator pattern. Specifically for Activators that extend ComponentActivatorAbstractBase.

• If the presented capabilities interest the ODL community, we can discuss extension of the infrastructure to support the new OSGI Bundle Activation patterns, that came along with MD-SAL

27

Page 28: Counters and Async Infrastructures · Counters and Async Infrastructures ... Multiple Counters [2016-02-16 15:18:27,626] ... • Using enum semantics instead of “register new counter”

Scope

28

Async Infra (Unwanted?)

Application A

OSGI Service OSGI Service

MD-SAL

Async Infra

Application B

OSGI Service OSGI ServiceAsync Infra

RPC / NotificationsRPC / Notifications

Page 29: Counters and Async Infrastructures · Counters and Async Infrastructures ... Multiple Counters [2016-02-16 15:18:27,626] ... • Using enum semantics instead of “register new counter”

Overview

29

• ServiceA implements PacketProcessingListener

Service A Service B Service CdoSomething() doSomethingElse()

onPacketReceived() Notification

MD-SAL

Application A

Page 30: Counters and Async Infrastructures · Counters and Async Infrastructures ... Multiple Counters [2016-02-16 15:18:27,626] ... • Using enum semantics instead of “register new counter”

public class ServiceA implements IServiceA, PacketProcessingListener {private IServiceB serviceB;

public void onPacketReceived(PacketReceived notification) {serviceB.doSomething();

}}

30

Page 31: Counters and Async Infrastructures · Counters and Async Infrastructures ... Multiple Counters [2016-02-16 15:18:27,626] ... • Using enum semantics instead of “register new counter”

public class ServiceB implements IServiceB {private IServiceC serviceC;

public void doSomething() {serviceC.doSomethingElse();

}}

31

Page 32: Counters and Async Infrastructures · Counters and Async Infrastructures ... Multiple Counters [2016-02-16 15:18:27,626] ... • Using enum semantics instead of “register new counter”

32

Service A Service B Service CdoSomething() doSomethingElse()

• The presented infrastructure enables you to convert these Java calls to Asynchronous

Multi-Threaded calls just by changing a configuration file!

Page 33: Counters and Async Infrastructures · Counters and Async Infrastructures ... Multiple Counters [2016-02-16 15:18:27,626] ... • Using enum semantics instead of “register new counter”

Example of the config-file/pools/OurThreadPool/poolSize=8

/handlers/com.hpe.ServiceB/PoolName=OurThreadPool

/handlers/com.hpe.ServiceB/WorkerCount=5

/handlers/com.hpe.ServiceC/PoolName=OurThreadPool

/handlers/com.hpe.ServiceC/WorkerCount=6

• We create a Thread Pool with 8 Threads, and define ServiceB and ServiceC to use it.

• ServiceB will use up to 5 concurrent Threads while ServiceC will use up to 6 concurrent Threads.

33

Page 34: Counters and Async Infrastructures · Counters and Async Infrastructures ... Multiple Counters [2016-02-16 15:18:27,626] ... • Using enum semantics instead of “register new counter”

Glimpse Under the Hood

34

ServiceAdoSomething() ServiceB Proxy

ServiceB Proxy ServiceBDedicated

Queue

Worker Thread

Worker Thread

Worker Thread

ServiceBdoSomething()

Page 35: Counters and Async Infrastructures · Counters and Async Infrastructures ... Multiple Counters [2016-02-16 15:18:27,626] ... • Using enum semantics instead of “register new counter”

Benefits• The developers can focus in coding the business logic of the Service.• The API remains exactly the same with the help of Proxies and Reflection.• Dynamically decide which services will be Async and control Thread Pools.• No need to write bug-prone Boilerplate Code for Async method invocation. This

includes BlockingQueues, Producer Threads, Consumer Threads, Executors, Thread Pools, synchronized blocks. Caveat: The code itself obviously must support multi-threaded execution, i.e. Concurrent data structures, etc.

• The Queue can be capped, and messages that exceed the cap will be dropped. Important for prevention of memory leaks.

35

Page 36: Counters and Async Infrastructures · Counters and Async Infrastructures ... Multiple Counters [2016-02-16 15:18:27,626] ... • Using enum semantics instead of “register new counter”

What if I want only part of the API to be ASYNC?

public class MyService implements IMyService {public void methodWeWantAsync() {}

public void anotherMethodWeWantAsync() {}

@SyncMethodpublic void methodWeWantSync() {}

}

@SyncMethod - Will tell the infrastructure to run this method using the Caller’s Thread36

Page 37: Counters and Async Infrastructures · Counters and Async Infrastructures ... Multiple Counters [2016-02-16 15:18:27,626] ... • Using enum semantics instead of “register new counter”

What about invoking a Service in a remote ODL?public class ServiceA implements IServiceA, PacketProcessingListener {

private IServiceB serviceB;

public void onPacketReceived(PacketReceived notification) {AsyncUtil.setTarget(new RemoteInstanceAddress(getIP()));serviceB.doSomething();

}private String getIP() {

if (…) {return “127.0.0.1”;

} else {return “1.2.3.4”;

}}

} 37

Page 38: Counters and Async Infrastructures · Counters and Async Infrastructures ... Multiple Counters [2016-02-16 15:18:27,626] ... • Using enum semantics instead of “register new counter”

Glimpse Under the Hood

38

ServiceAdoSomething() ServiceB

Proxy

ServiceBProxy

RemoteAsyncService

call(ip, service, msg)

RemoteAsyncService

AKKA Sendertell(serializedMsg)

ODL 1

Page 39: Counters and Async Infrastructures · Counters and Async Infrastructures ... Multiple Counters [2016-02-16 15:18:27,626] ... • Using enum semantics instead of “register new counter”

Glimpse Under the Hood

39

doSomething() ServiceB Proxy

ServiceB Proxy ServiceBDedicated

Queue

Worker Thread

Worker Thread

Worker Thread

ServiceBdoSomething()

AKKA Receiver

ODL 2

Page 40: Counters and Async Infrastructures · Counters and Async Infrastructures ... Multiple Counters [2016-02-16 15:18:27,626] ... • Using enum semantics instead of “register new counter”

How Bi-Directional Invocations Work?public class ServiceA implements IServiceA, PacketProcessingListener,

IServiceBRepliesHandler {

private IServiceB serviceB;

public void onPacketReceived(PacketReceived notification) {AsyncUtil.setTarget(new RemoteInstanceAddress(getIP()));serviceB.doSomething();

}

public void replyFromServiceB(Reply reply) {// Do logic

}} 40

Page 41: Counters and Async Infrastructures · Counters and Async Infrastructures ... Multiple Counters [2016-02-16 15:18:27,626] ... • Using enum semantics instead of “register new counter”

public class ServiceB implements IServiceB {private IServiceC serviceC;private IServiceBRepliesHandler replyHandler;

public void doSomething() {serviceC.doSomethingElse();AsyncUtil.setTarget(AsyncUtil.getCaller());replyHandler.replyFromServiceB(new Reply());

}}

41

Page 42: Counters and Async Infrastructures · Counters and Async Infrastructures ... Multiple Counters [2016-02-16 15:18:27,626] ... • Using enum semantics instead of “register new counter”

Caveats of the Bi-Directional Invocations• By keeping the code clean from Async Java utils, we lose their potential

capabilities. If the API of ServiceB would have been:

Future doSomething();

We could use Future.get(long Timeout), or Future.isDone(), etc.

• Reply code must contain the AsyncUtil.setTarget(AsyncUtil.getCaller()), but this may be resolved in the future using Marker Annotation on the Reply Interface. Currently, this call looks like “magic” for a developer that doesn’t know the underlying infrastructure.

42

Page 43: Counters and Async Infrastructures · Counters and Async Infrastructures ... Multiple Counters [2016-02-16 15:18:27,626] ... • Using enum semantics instead of “register new counter”

Under the Hood• When a Service registers using the BundleActivator, it must extend our new Activator:

AsyncAwareActivator. Example of Registration:

public class Activator extends AsyncAwareActivator {

protected void registerServices() {

registerNonAsyncService(SubscriberRegistrarDirect.class.getSimpleName(), SubscriberRegistrarDirect.class, //

impls(impl(ISubscriberRegistrar.class),

impl(ISystemLifecycle.class)), //

dep(IDatabaseService.class), //

dep(IGdms.class), //

dep(ISubscriberGlobalControllerCurator.class));}

}

43

Page 44: Counters and Async Infrastructures · Counters and Async Infrastructures ... Multiple Counters [2016-02-16 15:18:27,626] ... • Using enum semantics instead of “register new counter”

Under the Hood• The infrastructure registers a Proxy to the Service in the OSGI Registry instead of the

Service itself.

• It creates a dedicated Queue<MethodCallMessage> for the Service’s Async Method Calls.

public class MethodCallMessage {

Method method;

Object[] args;

}

These fields are populated from the Proxy call of java.lang.reflect.InvocationHandler: Object invoke(Object proxy, Method method, Object[] args)

44

Page 45: Counters and Async Infrastructures · Counters and Async Infrastructures ... Multiple Counters [2016-02-16 15:18:27,626] ... • Using enum semantics instead of “register new counter”

Under the Hood

• It also create dedicated WorkerThreads and assign them

to a ThreadPool according to the configuration file shown

previously.

• The WorkerThreads poll() the

Queue<MethodCallMessage> in a loop, and on each item

polled they invoke: method.invoke(handlerInstance, args)

45

Page 46: Counters and Async Infrastructures · Counters and Async Infrastructures ... Multiple Counters [2016-02-16 15:18:27,626] ... • Using enum semantics instead of “register new counter”

Under the Hood - Summary

• Using the illustrated capabilities, we harvest Java’s Proxy

and Reflection power, to allow a seamless conversion of

normal method invocation to Async and potentially Remote

invocation.

46

Page 47: Counters and Async Infrastructures · Counters and Async Infrastructures ... Multiple Counters [2016-02-16 15:18:27,626] ... • Using enum semantics instead of “register new counter”

Follow-Up Discussions

• Adaptions of the infrastructure to fit the community needs.

• Absence of capabilities in the current MD-SAL Async

solution

• Allow customization of Thread Pools. Applications that want to

listen to Notifications on a specific ThreadPool. Applications that

want its RPCs to run on a specific ThreadPool.

47

Page 48: Counters and Async Infrastructures · Counters and Async Infrastructures ... Multiple Counters [2016-02-16 15:18:27,626] ... • Using enum semantics instead of “register new counter”

Questions?

48