![Page 1: OSG(A)I: BECAUSE A.I. NEEDS A RUNTIME · A WAY TO EXECUTE MODELS IN OSGI A modular deep learning framework developed in OSGi Neural networks, operators, datasets, learners, … as](https://reader034.vdocuments.site/reader034/viewer/2022042222/5ec826c21451cc3fd737aa06/html5/thumbnails/1.jpg)
PUBLIC1
OSG(A)I: BECAUSE A.I. NEEDS A RUNTIME
TIM VERBELEN
Senior Researcherimec – Ghent University
![Page 2: OSG(A)I: BECAUSE A.I. NEEDS A RUNTIME · A WAY TO EXECUTE MODELS IN OSGI A modular deep learning framework developed in OSGi Neural networks, operators, datasets, learners, … as](https://reader034.vdocuments.site/reader034/viewer/2022042222/5ec826c21451cc3fd737aa06/html5/thumbnails/2.jpg)
PUBLIC
WHAT IS A.I.?
2
THE NEXT BIG THING (AFTER BLOCKCHAIN OFC!)
![Page 3: OSG(A)I: BECAUSE A.I. NEEDS A RUNTIME · A WAY TO EXECUTE MODELS IN OSGI A modular deep learning framework developed in OSGi Neural networks, operators, datasets, learners, … as](https://reader034.vdocuments.site/reader034/viewer/2022042222/5ec826c21451cc3fd737aa06/html5/thumbnails/3.jpg)
PUBLIC
WHAT IS MACHINE LEARNING?
3
LEARNING A FUNCTION APPROXIMATION MAPPING INPUTS TO OUTPUTS
ModelModel outputsinputs
![Page 4: OSG(A)I: BECAUSE A.I. NEEDS A RUNTIME · A WAY TO EXECUTE MODELS IN OSGI A modular deep learning framework developed in OSGi Neural networks, operators, datasets, learners, … as](https://reader034.vdocuments.site/reader034/viewer/2022042222/5ec826c21451cc3fd737aa06/html5/thumbnails/4.jpg)
PUBLIC
WHAT IS DEEP LEARNING?
4
USING DEEP NEURAL NETWORKS AS MACHINE LEARNING MODEL
Neural NetworkNeural Network
“cat”
![Page 5: OSG(A)I: BECAUSE A.I. NEEDS A RUNTIME · A WAY TO EXECUTE MODELS IN OSGI A modular deep learning framework developed in OSGi Neural networks, operators, datasets, learners, … as](https://reader034.vdocuments.site/reader034/viewer/2022042222/5ec826c21451cc3fd737aa06/html5/thumbnails/5.jpg)
PUBLIC
DEEP LEARNING FRAMEWORKS AND TOOLS
5
![Page 6: OSG(A)I: BECAUSE A.I. NEEDS A RUNTIME · A WAY TO EXECUTE MODELS IN OSGI A modular deep learning framework developed in OSGi Neural networks, operators, datasets, learners, … as](https://reader034.vdocuments.site/reader034/viewer/2022042222/5ec826c21451cc3fd737aa06/html5/thumbnails/6.jpg)
PUBLIC
THE AVERAGE DATA SCIENTIST WORKFLOW
1. Inspect and clean up the data
2. Select and encode features / outputs
3. Script together a model training procedure
4. Find some good hyperparameters
5. Dump the trained model
6
![Page 7: OSG(A)I: BECAUSE A.I. NEEDS A RUNTIME · A WAY TO EXECUTE MODELS IN OSGI A modular deep learning framework developed in OSGi Neural networks, operators, datasets, learners, … as](https://reader034.vdocuments.site/reader034/viewer/2022042222/5ec826c21451cc3fd737aa06/html5/thumbnails/7.jpg)
PUBLIC
BUT WHAT ABOUT PRODUCTION?
7
![Page 8: OSG(A)I: BECAUSE A.I. NEEDS A RUNTIME · A WAY TO EXECUTE MODELS IN OSGI A modular deep learning framework developed in OSGi Neural networks, operators, datasets, learners, … as](https://reader034.vdocuments.site/reader034/viewer/2022042222/5ec826c21451cc3fd737aa06/html5/thumbnails/8.jpg)
PUBLIC
TENSORFLOW SERVING
8
ClientServableHandler
DynamicManager
VersionPolicy
models/1/
…2/
…
Server
gRPC/REST
request
![Page 9: OSG(A)I: BECAUSE A.I. NEEDS A RUNTIME · A WAY TO EXECUTE MODELS IN OSGI A modular deep learning framework developed in OSGi Neural networks, operators, datasets, learners, … as](https://reader034.vdocuments.site/reader034/viewer/2022042222/5ec826c21451cc3fd737aa06/html5/thumbnails/9.jpg)
PUBLIC
CHECK THE BOXES
I want to query my models with an RPC/REST call
I train my models using TensorFlow
I deploy my models on a containerized infrastructure
I don’t need additional metadata/versioning besides a single incrementing integer
9
![Page 10: OSG(A)I: BECAUSE A.I. NEEDS A RUNTIME · A WAY TO EXECUTE MODELS IN OSGI A modular deep learning framework developed in OSGi Neural networks, operators, datasets, learners, … as](https://reader034.vdocuments.site/reader034/viewer/2022042222/5ec826c21451cc3fd737aa06/html5/thumbnails/10.jpg)
PUBLIC
OSGI CAN HELP!
A unit of deployment Package ML model as an OSGi bundle
A resolveable artifact with requirements and capabilities Requirements: what do you need to run this ML model Capabilities: what kind of inputs can it process and what kind of outputs
does it give you? A lightweight service model
Access your model via an OSGi service
Service selection at runtime Use service properties and target filters to select the best model at runtime
10
![Page 11: OSG(A)I: BECAUSE A.I. NEEDS A RUNTIME · A WAY TO EXECUTE MODELS IN OSGI A modular deep learning framework developed in OSGi Neural networks, operators, datasets, learners, … as](https://reader034.vdocuments.site/reader034/viewer/2022042222/5ec826c21451cc3fd737aa06/html5/thumbnails/11.jpg)
PUBLIC
A WAY TO SHARE ML MODELS
A common format for describing computation graphs
defined as protocol buffers specifies data types and operators framework agnostic
11
OPEN NEURAL NETWORK EXCHANGE (ONNX)
![Page 12: OSG(A)I: BECAUSE A.I. NEEDS A RUNTIME · A WAY TO EXECUTE MODELS IN OSGI A modular deep learning framework developed in OSGi Neural networks, operators, datasets, learners, … as](https://reader034.vdocuments.site/reader034/viewer/2022042222/5ec826c21451cc3fd737aa06/html5/thumbnails/12.jpg)
PUBLIC
A WAY TO EXECUTE MODELS IN OSGI
A modular deep learning framework developed in OSGi Neural networks, operators, datasets, learners, … as an OSGi service Low-level operations via JNI and blas/cublas/cudnn backends Distributed deployments using OSGi remote services Web UI to build, deploy, train models
12
DIANNE FRAMEWORK
http://dianne.intec.ugent.be
![Page 13: OSG(A)I: BECAUSE A.I. NEEDS A RUNTIME · A WAY TO EXECUTE MODELS IN OSGI A modular deep learning framework developed in OSGi Neural networks, operators, datasets, learners, … as](https://reader034.vdocuments.site/reader034/viewer/2022042222/5ec826c21451cc3fd737aa06/html5/thumbnails/13.jpg)
PUBLIC
DEMO TIME
13
![Page 14: OSG(A)I: BECAUSE A.I. NEEDS A RUNTIME · A WAY TO EXECUTE MODELS IN OSGI A modular deep learning framework developed in OSGi Neural networks, operators, datasets, learners, … as](https://reader034.vdocuments.site/reader034/viewer/2022042222/5ec826c21451cc3fd737aa06/html5/thumbnails/14.jpg)
PUBLIC
NOW LET’S ADD A ROBOT
14
![Page 15: OSG(A)I: BECAUSE A.I. NEEDS A RUNTIME · A WAY TO EXECUTE MODELS IN OSGI A modular deep learning framework developed in OSGi Neural networks, operators, datasets, learners, … as](https://reader034.vdocuments.site/reader034/viewer/2022042222/5ec826c21451cc3fd737aa06/html5/thumbnails/15.jpg)
PUBLIC
NOW LET’S ADD A ROBOT
15
![Page 16: OSG(A)I: BECAUSE A.I. NEEDS A RUNTIME · A WAY TO EXECUTE MODELS IN OSGI A modular deep learning framework developed in OSGi Neural networks, operators, datasets, learners, … as](https://reader034.vdocuments.site/reader034/viewer/2022042222/5ec826c21451cc3fd737aa06/html5/thumbnails/16.jpg)
PUBLIC
PUTTING THINGS TOGETHER
16
@Reference Camera camera;@Reference NeuralNetwork nn;@Reference ArmController controller;
public Promise<Boolean> grasp(){ return camera.stream() // returns PushStream<Frame>
// timeout after 10 seconds.timeout(Duration.ofSeconds(10))// process camera frame with neural network.map(frame -> nn.forward(toTensor(frame))// use result to update controller.map(r -> controller.update(r))// short circuit in case of success.anyMatch(s -> s);
}
![Page 17: OSG(A)I: BECAUSE A.I. NEEDS A RUNTIME · A WAY TO EXECUTE MODELS IN OSGI A modular deep learning framework developed in OSGi Neural networks, operators, datasets, learners, … as](https://reader034.vdocuments.site/reader034/viewer/2022042222/5ec826c21451cc3fd737aa06/html5/thumbnails/17.jpg)
PUBLIC
NOW LET’S ADD A ROBOT
17
![Page 18: OSG(A)I: BECAUSE A.I. NEEDS A RUNTIME · A WAY TO EXECUTE MODELS IN OSGI A modular deep learning framework developed in OSGi Neural networks, operators, datasets, learners, … as](https://reader034.vdocuments.site/reader034/viewer/2022042222/5ec826c21451cc3fd737aa06/html5/thumbnails/18.jpg)
PUBLIC18
![Page 19: OSG(A)I: BECAUSE A.I. NEEDS A RUNTIME · A WAY TO EXECUTE MODELS IN OSGI A modular deep learning framework developed in OSGi Neural networks, operators, datasets, learners, … as](https://reader034.vdocuments.site/reader034/viewer/2022042222/5ec826c21451cc3fd737aa06/html5/thumbnails/19.jpg)
PUBLIC19
THANK YOU!
http://dianne.intec.ugent.be