real time systems compilation with lopht dumitru potop-butucaru inria paris/ aoste team jan. 2016 1
DESCRIPTION
Criticality Allocation Partitioning Other real-time Embedded control systems 3 Continuous-time model Functional specification Platform-independent real-time specification Controller discretization Other non-functional requirements Platform model (HW+OS) Other engineering fields (HW design, reliability, etc…) Plant+Controller Computer engineering Non-functional specification Cyclic controlPeriods Deadlines Latency constraints Implementation Executable implementation Functional correctness CPU, bus, etc. Topology WCET/WCCT OS/drivers Control engineering Respect of requirementsTRANSCRIPT
![Page 1: Real Time Systems Compilation with Lopht Dumitru Potop-Butucaru INRIA Paris/ AOSTE team Jan. 2016 1](https://reader035.vdocuments.site/reader035/viewer/2022062504/5a4d1b6d7f8b9ab0599b3f52/html5/thumbnails/1.jpg)
1
Real Time Systems Compilation with Lopht
Dumitru Potop-ButucaruINRIA Paris/ AOSTE team
Jan. 2016
![Page 2: Real Time Systems Compilation with Lopht Dumitru Potop-Butucaru INRIA Paris/ AOSTE team Jan. 2016 1](https://reader035.vdocuments.site/reader035/viewer/2022062504/5a4d1b6d7f8b9ab0599b3f52/html5/thumbnails/2.jpg)
Embedded control systems• Computing system that controls the execution of a piece of
« physical » equipment, in order to ensure its correct functioning
2
Plant(controlled
« physical » system)
Embedded control system
SensorsActuatorsAnalog domain
Digital domain
![Page 3: Real Time Systems Compilation with Lopht Dumitru Potop-Butucaru INRIA Paris/ AOSTE team Jan. 2016 1](https://reader035.vdocuments.site/reader035/viewer/2022062504/5a4d1b6d7f8b9ab0599b3f52/html5/thumbnails/3.jpg)
3
CriticalityAllocationPartitioningOther real-time
Embedded control systemsContinuous-time
model
Functional specification
Platform-independentreal-time specification
Controller discretization
Other non-functional requirements
Platform model
(HW+OS)
Other engineering fields(HW design, reliability, etc…)
Plant+Controller
Com
pute
r eng
inee
ring
Non-functional specificationCyclic control PeriodsDeadlinesLatency constraints
Implementation
Executable implementation
Functionalcorrectness
CPU, bus, etc.TopologyWCET/WCCTOS/drivers
Cont
rol
engi
neer
ing
Respect ofrequirements
![Page 4: Real Time Systems Compilation with Lopht Dumitru Potop-Butucaru INRIA Paris/ AOSTE team Jan. 2016 1](https://reader035.vdocuments.site/reader035/viewer/2022062504/5a4d1b6d7f8b9ab0599b3f52/html5/thumbnails/4.jpg)
4
Embedded systems implementation
• Still largely a craft in the industry– Important manual and/or unformalized phases
• Some that could be automatized with existing industry-grade tools (modulo certification arguments)– Discretization of the continuous-time model – Construction of tasks from pieces of C code– Allocation and scheduling of tasks for functional correctness– Programming and implementing complex timed behaviors
• Some that cannot (to my knowledge)– Building platform and implementation models for precise and safe timing analysis – Allocation & scheduling providing hard guarantees of respect for real-time and
other non-functional requirements
– Resource-consuming, error-prone– Ultimately rely on test to check system correctness
![Page 5: Real Time Systems Compilation with Lopht Dumitru Potop-Butucaru INRIA Paris/ AOSTE team Jan. 2016 1](https://reader035.vdocuments.site/reader035/viewer/2022062504/5a4d1b6d7f8b9ab0599b3f52/html5/thumbnails/5.jpg)
5
Hand-crafting implementations no longer scales
• Complexity explodes– Large numbers of resources (multi-/many-cores)– Large amount of SW (more tasks, more system SW)– More complex hardware, OS, SW
• Communication networks (NoCs, AFDX, etc.), hierarchical schedulers, dependent tasks
• Efficiency becomes critical– Parallelized filters = dependent task systems
• Can rapidly lose performance if multi-processor scheduling is not efficient– Performance highly depends on mapping, need to fine tune all system
parameters• Allocation, scheduling• Memory allocations• Synchronizations, communications…
• Delegating mapping to HW/OS results in impredictability– Cache coherency, GPUs, dynamic allocation and scheduling, etc.
![Page 6: Real Time Systems Compilation with Lopht Dumitru Potop-Butucaru INRIA Paris/ AOSTE team Jan. 2016 1](https://reader035.vdocuments.site/reader035/viewer/2022062504/5a4d1b6d7f8b9ab0599b3f52/html5/thumbnails/6.jpg)
6
Off-line automation is needed
• Requires full formalization:– Functional specification– Non-functional requirements– Execution platform models
• Sound abstraction issue – Formalization of implementation correctness
• Requires efficient algorithms for analysis and synthesis
Standardization is a prerequisite(allows cost-effective tool-building)
![Page 7: Real Time Systems Compilation with Lopht Dumitru Potop-Butucaru INRIA Paris/ AOSTE team Jan. 2016 1](https://reader035.vdocuments.site/reader035/viewer/2022062504/5a4d1b6d7f8b9ab0599b3f52/html5/thumbnails/7.jpg)
A success story of off-line automation: compilation
• Low level of the embedded implementation flow.• Made possible by the early standardization of
programming languages, and execution platforms (ISAs, ABIs, APIs).
• Almost complete replacement of assembly coding, even in embedded systems design.
7
C/Ada/… programs
Compiler
Executable implementation
Platform model(ISA+ABI+API+ti
ming)
![Page 8: Real Time Systems Compilation with Lopht Dumitru Potop-Butucaru INRIA Paris/ AOSTE team Jan. 2016 1](https://reader035.vdocuments.site/reader035/viewer/2022062504/5a4d1b6d7f8b9ab0599b3f52/html5/thumbnails/8.jpg)
Real-time scheduling vs. compilationFunctional
specificationPlatform-independentreal-time specification
Other non-functional requirements
High-levelplatform
model
Real time scheduling
Abstract implementation model
8
C/Ada/… programs
Compiler
Executable implementation
Off-line vs. On-line(RM/EDF/RTC/etc.)
Scheduling tablesTask models
R-T calculus models
Platform model(ISA+ABI+API+ti
ming)
Real
-tim
e sc
hedu
ling
Com
pila
tion
![Page 9: Real Time Systems Compilation with Lopht Dumitru Potop-Butucaru INRIA Paris/ AOSTE team Jan. 2016 1](https://reader035.vdocuments.site/reader035/viewer/2022062504/5a4d1b6d7f8b9ab0599b3f52/html5/thumbnails/9.jpg)
Real-time scheduling vs. compilationFunctional
specificationPlatform-independentreal-time specification
Other non-functional requirements
High-levelplatform
model
Real time scheduling
Abstract implementation model
9
C/Ada/… programs
Compiler
Executable implementation
Platform model(ISA+ABI+API+ti
ming)
Real
-tim
e sc
hedu
ling
Com
pila
tion
Abstraction layer (WCET analysis, protocol synthesis, code generation) Rarely sound
![Page 10: Real Time Systems Compilation with Lopht Dumitru Potop-Butucaru INRIA Paris/ AOSTE team Jan. 2016 1](https://reader035.vdocuments.site/reader035/viewer/2022062504/5a4d1b6d7f8b9ab0599b3f52/html5/thumbnails/10.jpg)
Challenge: full automationFunctional
specificationPlatform-independentreal-time specification
Other non-functional requirements
High-levelplatform
model
Real time scheduling
Abstract implementation model
10
C/Ada/… programs
Compiler
Executable implementation
Platform model(ISA+ABI+API+ti
ming)
Abstraction layer (WCET analysis, protocol synthesis, code generation) Rarely sound
Real
-tim
e sc
hedu
ling
Com
pila
tion
![Page 11: Real Time Systems Compilation with Lopht Dumitru Potop-Butucaru INRIA Paris/ AOSTE team Jan. 2016 1](https://reader035.vdocuments.site/reader035/viewer/2022062504/5a4d1b6d7f8b9ab0599b3f52/html5/thumbnails/11.jpg)
11
Challenge: full automation• Historically: difficult, due to lack of standardization
• Recently: functional and non-functional specification languages– Dynamic systems specification languages:
• Simulink, LabView, Scade, etc.• Code generation ensuring functional correctness, not schedulability• Discretization
– Systems engineering languages: • SysML, AADL… • Real-time, Criticality/partitioning, Allocation…
• Today: Execution platforms with support for real-time:– ARINC 653, AUTOSAR, TTP/TTEthernet, etc.– Many-cores (some of them)
![Page 12: Real Time Systems Compilation with Lopht Dumitru Potop-Butucaru INRIA Paris/ AOSTE team Jan. 2016 1](https://reader035.vdocuments.site/reader035/viewer/2022062504/5a4d1b6d7f8b9ab0599b3f52/html5/thumbnails/12.jpg)
12
Real time systems compilation• Move from specification to running implementation
– Fully automatically• No human intervention during the compilation itself
– Inputs and outputs must all be formalized • Functional and non-functional inputs, (model of) generated code• Provide formal correctness guarantees (functional&non-functional) • No break in the formal chain
– Formal proof possible
– Failure traceability • Functional and non-functional causes• Easy to do trial-and-error systems design
– Fast and efficient• Fine-grain platform and application modeling• Use of fast heuristics
– Compilation, real time scheduling, synchronous languages, etc.– Exact techniques (e.g. SMT-based) do not scale [FORMATS’15]
![Page 13: Real Time Systems Compilation with Lopht Dumitru Potop-Butucaru INRIA Paris/ AOSTE team Jan. 2016 1](https://reader035.vdocuments.site/reader035/viewer/2022062504/5a4d1b6d7f8b9ab0599b3f52/html5/thumbnails/13.jpg)
13
Real time systems compilation
• Start with statically scheduled systems• Timing analysis-friendly (if HW&OS are also friendly)
– Good performance and timing predictability– Checking functional correctness and computing WCET/WCRT is easy
using existing tools– We have good experience with static scheduling approaches
• Classes of applications with practical importance– Critical embedded control applications– Signal/image processing
• Implementation model: scheduling/reservation tables– Shared between real-time scheduling and compilation
» Facilitate reuse of algorithms
![Page 14: Real Time Systems Compilation with Lopht Dumitru Potop-Butucaru INRIA Paris/ AOSTE team Jan. 2016 1](https://reader035.vdocuments.site/reader035/viewer/2022062504/5a4d1b6d7f8b9ab0599b3f52/html5/thumbnails/14.jpg)
14
Previous work• Work on superscalar/VLIW/many-core compilation
– Optimization, not respect of requirements – No hard real-time guarantees – Finer level of control
• Work on off-line real-time scheduling– AAA/SynDEx (Sorel et al.)
• Does not have: classical compiler optimizations (pipelining), fine-grain execution condition analysis, time triggered code generation, partitioning, preemptable tasks, memory allocation…
– Simulink -> Scade -> TTA (Caspi et al.)• Does not have: Automatic allocation, conditional execution, preemption, mapping
optimizations (pipelining, memory bank allocation, etc.).– Prelude (Forget et al.)
• Does not have: Partitioning, deadlines longer than periods, memory bank allocations.– PsyC/PharOS
• Input is lower-level (can be used as back-end to our approach)
![Page 15: Real Time Systems Compilation with Lopht Dumitru Potop-Butucaru INRIA Paris/ AOSTE team Jan. 2016 1](https://reader035.vdocuments.site/reader035/viewer/2022062504/5a4d1b6d7f8b9ab0599b3f52/html5/thumbnails/15.jpg)
15
The real time compiler Lopht• Compiler for statically-scheduled real-time systems
– Functional specification: Data-flow synchronous (e.g. Lustre/Scade)– Target execution platforms:
• Distributed time-triggered systems– ARINC 653-based processors, TTEthernet – Full generation of ARINC 653 configuration & partition C/APEX code & bus schedule
• « WCET-friendly » many-cores (our own, Kalray MPPA)– Full generation of C bare metal code, including communication and synchronization operations,
memory allocations, NoC configuration, etc.– WCET analysis of generated parallel code, including synchronization code (sound architecture
abstraction layer)
– Meaningful applications:• Space launcher model (Airbus DS)
– Complex non-functional requirements• Passenger exchange (Alstom/IRT SystemX)
– Focus on mixed criticality• Platooning application model (CyCab)
– Parallelized image processing code inside a control loop
![Page 16: Real Time Systems Compilation with Lopht Dumitru Potop-Butucaru INRIA Paris/ AOSTE team Jan. 2016 1](https://reader035.vdocuments.site/reader035/viewer/2022062504/5a4d1b6d7f8b9ab0599b3f52/html5/thumbnails/16.jpg)
16
The principle: table-based scheduling
• Example: Engine ignition application
– Functional specification: dataflow synchronous• Cyclic execution model• 4 modes determined by 2 independent switches
HS_IN
ID
F3
IDG
if (HS)
if (not HS)ID
F2
F1
FS_IN
M
N
if (FS)
if (not FS)
ID
Vx
Δ HS: High speed modeFS: Fail-safe mode
![Page 17: Real Time Systems Compilation with Lopht Dumitru Potop-Butucaru INRIA Paris/ AOSTE team Jan. 2016 1](https://reader035.vdocuments.site/reader035/viewer/2022062504/5a4d1b6d7f8b9ab0599b3f52/html5/thumbnails/17.jpg)
17
The principle: table-based scheduling
• Example : Non-functional specification– 3 CPUs, 1 broadcast bus
– WCETs/WCCTs– Allocation constraints
P0 P1BUS
read_input=1F1 = 3F2 = 8F3 = 5
F1 = 3F2 = 8F3 = 5G = 3Bool=2
ID=5V=2x=2
17
P2
F1 = 3F2 = 8F3 = 2N = 3M = 1
![Page 18: Real Time Systems Compilation with Lopht Dumitru Potop-Butucaru INRIA Paris/ AOSTE team Jan. 2016 1](https://reader035.vdocuments.site/reader035/viewer/2022062504/5a4d1b6d7f8b9ab0599b3f52/html5/thumbnails/18.jpg)
The principle: table-based scheduling
• List scheduling (like in compilers, SynDEx, etc.)– Allocate and schedule the blocks 1 by 1– Allocation and scheduling are optimal for each block
• Criterion: Cost function– All legal allocations are tried, we retain the one that minimizes the cost
function– Cost functions:
» Simplest: worst-case end date of the block (this example)» More complex can include e.g. locality-related terms
• Does not imply global optimality (heuristic)– No backtracking
– Communication mapping is done during block mapping– Failure reporting: current scheduling status, blocking reason18
![Page 19: Real Time Systems Compilation with Lopht Dumitru Potop-Butucaru INRIA Paris/ AOSTE team Jan. 2016 1](https://reader035.vdocuments.site/reader035/viewer/2022062504/5a4d1b6d7f8b9ab0599b3f52/html5/thumbnails/19.jpg)
The principle: table-based scheduling
• Result: conditioned scheduling table
19
012345
P0 P1 P2temps
67891011
1314
12
1516
BusHS_INFS_IN
if(not HS)F1
if(not HS)F2
if(FS)N
send(P0,FS)
if(not HS) F3
if(not HS)send(P0,V)
send(P0,HS)
if(HS)G
if(not FS)M
if(not HS& not FS)
send(P0,ID)if(HS&
not FS)send(P1,ID)
![Page 20: Real Time Systems Compilation with Lopht Dumitru Potop-Butucaru INRIA Paris/ AOSTE team Jan. 2016 1](https://reader035.vdocuments.site/reader035/viewer/2022062504/5a4d1b6d7f8b9ab0599b3f52/html5/thumbnails/20.jpg)
20
Before scheduling: flattening the dataflow
HS_IN
ID
F3
IDG
if (HS)
if (not HS)ID
F2
F1
FS_IN
M
N
if (FS)
if (not FS)
IDx
Init=v0HS: High speed modeFS: Fail-safe mode
V
• Translation to non-hierarchic data-flow
![Page 21: Real Time Systems Compilation with Lopht Dumitru Potop-Butucaru INRIA Paris/ AOSTE team Jan. 2016 1](https://reader035.vdocuments.site/reader035/viewer/2022062504/5a4d1b6d7f8b9ab0599b3f52/html5/thumbnails/21.jpg)
21
Before scheduling: flattening the dataflow
• Translation to non-hierarchic data-flow
HS_IN G
FS_IN
M
HS: High speed modeFS: Fail-safe mode
N
true
true HSID
FS
FSHS
F2
F1 x
Init=v0
F3V
HSHS
HS
HS FS
HS FS
HS
HS
HSHS
![Page 22: Real Time Systems Compilation with Lopht Dumitru Potop-Butucaru INRIA Paris/ AOSTE team Jan. 2016 1](https://reader035.vdocuments.site/reader035/viewer/2022062504/5a4d1b6d7f8b9ab0599b3f52/html5/thumbnails/22.jpg)
Scheduling algorithm
• Step 1: order the dataflow blocks (« list scheduling »)– Total order compatible with the data dependencies
(except for delayed ones)
22
HS_IN G
FS_IN
M
N
true
true HSID
FS
FSHS
F2
F1 x
F3V
HSHS
HS FS
HS FS
HSHS
![Page 23: Real Time Systems Compilation with Lopht Dumitru Potop-Butucaru INRIA Paris/ AOSTE team Jan. 2016 1](https://reader035.vdocuments.site/reader035/viewer/2022062504/5a4d1b6d7f8b9ab0599b3f52/html5/thumbnails/23.jpg)
Scheduling algorithm
• Step 1: order the dataflow blocks (« list scheduling »)– Total order compatible with the data dependencies
(except for delayed ones)
23
HS_IN G
FS_IN
M
N
true
true HSID
FS
FSHS
F2
F1 x
F3V
HSHS
HS FS
HS FS
HSHS
2
1
3
4 6
7 5
8
![Page 24: Real Time Systems Compilation with Lopht Dumitru Potop-Butucaru INRIA Paris/ AOSTE team Jan. 2016 1](https://reader035.vdocuments.site/reader035/viewer/2022062504/5a4d1b6d7f8b9ab0599b3f52/html5/thumbnails/24.jpg)
Scheduling algorithm
• Step 2: allocate and schedule the blocks 1 by 1– Allocation and scheduling are optimal for each block,
taken separately.• Criterion: Cost function
– Try all legal allocations, but retain only one of those that minimize the cost function
– In this example, cost function = reservation end date• Local optimality does not imply global optimality
– No backtracking
– Communication scheduling is done on demand, during block scheduling.• Multiple routes possible => more complicated 24
![Page 25: Real Time Systems Compilation with Lopht Dumitru Potop-Butucaru INRIA Paris/ AOSTE team Jan. 2016 1](https://reader035.vdocuments.site/reader035/viewer/2022062504/5a4d1b6d7f8b9ab0599b3f52/html5/thumbnails/25.jpg)
Scheduling algorithm
• Step 2: allocate and schedule the blocks 1 by 1
25
012345
P0 P1 P2time
67891011
1314
12
1516
Bus
At first, the scheduling table is empty.
![Page 26: Real Time Systems Compilation with Lopht Dumitru Potop-Butucaru INRIA Paris/ AOSTE team Jan. 2016 1](https://reader035.vdocuments.site/reader035/viewer/2022062504/5a4d1b6d7f8b9ab0599b3f52/html5/thumbnails/26.jpg)
Scheduling algorithm
• Step 2: allocate and schedule the blocks 1 by 1
26
012345
P0 P1 P2time
67891011
1314
12
1516
BusHS_IN
Operation HS_IN can only be allocated on P0. The optimal schedule places the operation at date 0.
![Page 27: Real Time Systems Compilation with Lopht Dumitru Potop-Butucaru INRIA Paris/ AOSTE team Jan. 2016 1](https://reader035.vdocuments.site/reader035/viewer/2022062504/5a4d1b6d7f8b9ab0599b3f52/html5/thumbnails/27.jpg)
Scheduling algorithm
• Step 2: allocate and schedule the blocks 1 by 1
27
012345
P0 P1 P2time
67891011
1314
12
1516
BusHS_IN
Operation FS_IN can only be allocated on P0. The optimal schedule places the operation at date 1.
FS_IN
![Page 28: Real Time Systems Compilation with Lopht Dumitru Potop-Butucaru INRIA Paris/ AOSTE team Jan. 2016 1](https://reader035.vdocuments.site/reader035/viewer/2022062504/5a4d1b6d7f8b9ab0599b3f52/html5/thumbnails/28.jpg)
Scheduling algorithm
• Step 2: allocate and schedule the blocks 1 by 1
28
012345
P0 P1 P2time
67891011
1314
12
1516
BusHS_IN
Operation F1 can be allocated on P0, P1, or P2. We retain the allocation on P0 (in darker blue) be-cause in this case F1 terminates at date 5, whereas on P1 or P2 it would end at date 6 (due to the communication of HS, needed to compute the activation condition).
FS_IN
if(not HS)F1
send(P0,HS)
if(not HS)F1
if(not HS)F1
![Page 29: Real Time Systems Compilation with Lopht Dumitru Potop-Butucaru INRIA Paris/ AOSTE team Jan. 2016 1](https://reader035.vdocuments.site/reader035/viewer/2022062504/5a4d1b6d7f8b9ab0599b3f52/html5/thumbnails/29.jpg)
Scheduling algorithm
• Step 2: allocate and schedule the blocks 1 by 1
29
012345
P0 P1 P2time
67891011
1314
12
1516
BusHS_INFS_IN
if(not HS)F1
Operation F1 can be allocated on P0, P1, or P2. We retain the allocation on P0 (in darker blue) be-cause in this case F1 terminates at date 5, whereas on P1 or P2 it would end at date 6 (due to the communication of HS, needed to compute the activation condition).
![Page 30: Real Time Systems Compilation with Lopht Dumitru Potop-Butucaru INRIA Paris/ AOSTE team Jan. 2016 1](https://reader035.vdocuments.site/reader035/viewer/2022062504/5a4d1b6d7f8b9ab0599b3f52/html5/thumbnails/30.jpg)
Scheduling algorithm
• Step 2: allocate and schedule the blocks 1 by 1
30
012345
P0 P1 P2time
67891011
1314
12
1516
BusHS_IN
Same for F2.FS_IN
if(not HS)F1
send(P0,HS)
if(not HS)F2
if(not HS)send(P0,x)
if(not HS)F2
if(not HS)F2
![Page 31: Real Time Systems Compilation with Lopht Dumitru Potop-Butucaru INRIA Paris/ AOSTE team Jan. 2016 1](https://reader035.vdocuments.site/reader035/viewer/2022062504/5a4d1b6d7f8b9ab0599b3f52/html5/thumbnails/31.jpg)
Scheduling algorithm
• Step 2: allocate and schedule the blocks 1 by 1
31
012345
P0 P1 P2time
67891011
1314
12
1516
BusHS_IN
Same for F2.FS_IN
if(not HS)F1
if(not HS)F2
![Page 32: Real Time Systems Compilation with Lopht Dumitru Potop-Butucaru INRIA Paris/ AOSTE team Jan. 2016 1](https://reader035.vdocuments.site/reader035/viewer/2022062504/5a4d1b6d7f8b9ab0599b3f52/html5/thumbnails/32.jpg)
Scheduling algorithm
• Step 2: allocate and schedule the blocks 1 by 1
32
012345
P0 P1 P2time
67891011
1314
12
1516
BusHS_IN
N can only be allocated on P2. This forces the transmission of FS.The 2 operations are scheduled as soon as possible (ASAP).
FS_IN
if(not HS)F1
if(not HS)F2
if(FS)N
send(P0,FS)
![Page 33: Real Time Systems Compilation with Lopht Dumitru Potop-Butucaru INRIA Paris/ AOSTE team Jan. 2016 1](https://reader035.vdocuments.site/reader035/viewer/2022062504/5a4d1b6d7f8b9ab0599b3f52/html5/thumbnails/33.jpg)
Scheduling algorithm
• Step 2: allocate and schedule the blocks 1 by 1
33
012345
P0 P1 P2time
67891011
1314
12
1516
BusHS_IN
F3 can be allocated on P0, P1, or P2. It is executed much faster on P2, which compensates for the needed communication time. The allocation on P2 is therefore retained.
FS_IN
if(not HS)F1
if(not HS)F2
if(FS)N
send(P0,FS)
if(not HS) F3
if(not HS)send(P0,V)
send(P0,HS)
if(not HS) F3
if(not HS) F3
![Page 34: Real Time Systems Compilation with Lopht Dumitru Potop-Butucaru INRIA Paris/ AOSTE team Jan. 2016 1](https://reader035.vdocuments.site/reader035/viewer/2022062504/5a4d1b6d7f8b9ab0599b3f52/html5/thumbnails/34.jpg)
Scheduling algorithm
• Step 2: allocate and schedule the blocks 1 by 1
34
012345
P0 P1 P2time
67891011
1314
12
1516
BusHS_IN
F3 can be allocated on P0, P1, or P2. It is executed much faster on P2, which compensates for the needed communication time. The allocation on P2 is therefore retained.
FS_IN
if(not HS)F1
if(not HS)F2
if(FS)N
send(P0,FS)
if(not HS) F3
if(not HS)send(P0,V)
send(P0,HS)
![Page 35: Real Time Systems Compilation with Lopht Dumitru Potop-Butucaru INRIA Paris/ AOSTE team Jan. 2016 1](https://reader035.vdocuments.site/reader035/viewer/2022062504/5a4d1b6d7f8b9ab0599b3f52/html5/thumbnails/35.jpg)
Scheduling algorithm
• Step 2: allocate and schedule the blocks 1 by 1
35
012345
P0 P1 P2time
67891011
1314
12
1516
BusHS_IN
G can be allocated on P0, P1, or P2. Allocation on P1 minimizes the end date, so it is retained.
FS_IN
if(not HS)F1
if(not HS)F2
if(FS)N
send(P0,FS)
if(not HS) F3
if(not HS)send(P0,V)
send(P0,HS)
if(HS)G if(HS)
G
if(HS)G
![Page 36: Real Time Systems Compilation with Lopht Dumitru Potop-Butucaru INRIA Paris/ AOSTE team Jan. 2016 1](https://reader035.vdocuments.site/reader035/viewer/2022062504/5a4d1b6d7f8b9ab0599b3f52/html5/thumbnails/36.jpg)
Scheduling algorithm
• Step 2: allocate and schedule the blocks 1 by 1
36
012345
P0 P1 P2time
67891011
1314
12
1516
BusHS_IN
G can be allocated on P0, P1, or P2. Allocation on P1 minimizes the end date, so it is retained.
FS_IN
if(not HS)F1
if(not HS)F2
if(FS)N
send(P0,FS)
if(not HS) F3
if(not HS)send(P0,V)
send(P0,HS)
if(HS)G
![Page 37: Real Time Systems Compilation with Lopht Dumitru Potop-Butucaru INRIA Paris/ AOSTE team Jan. 2016 1](https://reader035.vdocuments.site/reader035/viewer/2022062504/5a4d1b6d7f8b9ab0599b3f52/html5/thumbnails/37.jpg)
Scheduling algorithm
• Step 2: allocate and schedule the blocks 1 by 1
37
012345
P0 P1 P2time
67891011
1314
12
1516
BusHS_IN
M can be allocated only on P2. Depen-ding on the value of HS, its input ID comes from either P0, or from P1. Also this data is not needed when M is not executed, hence the execution conditions of the send operations.
FS_IN
if(not HS)F1
if(not HS)F2
if(FS)N
send(P0,FS)
if(not HS) F3
if(not HS)send(P0,V)
send(P0,HS)
if(HS)G
if(not FS)M
if(not HS& not FS)
send(P0,ID)if(HS&
not FS)send(P1,ID)
![Page 38: Real Time Systems Compilation with Lopht Dumitru Potop-Butucaru INRIA Paris/ AOSTE team Jan. 2016 1](https://reader035.vdocuments.site/reader035/viewer/2022062504/5a4d1b6d7f8b9ab0599b3f52/html5/thumbnails/38.jpg)
Scheduling algorithm
• Step 2: allocate and schedule the blocks 1 by 1
38
012345
P0 P1 P2time
67891011
1314
12
1516
BusHS_IN
Generated schedule:- Latency: 17 - Throughput:
1/17
FS_IN
if(not HS)F1
if(not HS)F2
if(FS)N
send(P0,FS)
if(not HS) F3
if(not HS)send(P0,V)
send(P0,HS)
if(HS)G
if(not FS)M
if(not HS& not FS)
send(P0,ID)if(HS&
not FS)send(P1,ID)
![Page 39: Real Time Systems Compilation with Lopht Dumitru Potop-Butucaru INRIA Paris/ AOSTE team Jan. 2016 1](https://reader035.vdocuments.site/reader035/viewer/2022062504/5a4d1b6d7f8b9ab0599b3f52/html5/thumbnails/39.jpg)
39
But this is not enough (1/3)
• General-purpose optimization, to reduce communications, reduce makespan, and increase throughput– Software pipelining of scheduling tables• Reduce throughput without changing makespan • Take into account conditional execution to allow double
reservation between successive cycles– Safe double reservation of resources• Precise analysis of execution conditions
![Page 40: Real Time Systems Compilation with Lopht Dumitru Potop-Butucaru INRIA Paris/ AOSTE team Jan. 2016 1](https://reader035.vdocuments.site/reader035/viewer/2022062504/5a4d1b6d7f8b9ab0599b3f52/html5/thumbnails/40.jpg)
But this is not enough (1/3)
• Example: software pipelining
40
012345
P0 P1 P2time
67891011
1314
12
1516
BusHS_IN
Generated schedule:- Latency: 17 - Throughput:
1/17
FS_IN
if(not HS)F1
if(not HS)F2
if(FS)N
send(P0,FS)
if(not HS) F3
if(not HS)send(P0,V)
send(P0,HS)
if(HS)G
if(not FS)M
if(not HS& not FS)
send(P0,ID)if(HS&
not FS)send(P1,ID)
![Page 41: Real Time Systems Compilation with Lopht Dumitru Potop-Butucaru INRIA Paris/ AOSTE team Jan. 2016 1](https://reader035.vdocuments.site/reader035/viewer/2022062504/5a4d1b6d7f8b9ab0599b3f52/html5/thumbnails/41.jpg)
But this is not enough (1/3)
• Example: software pipelining
41
012345
P0 P1 P2time
67891011
1314
12
1516
BusHS_IN
Generated schedule:- Latency: 17 - Throughput:
1/17
FS_IN
if(not HS)F1
if(not HS)F2
if(FS)N
send(P0,FS)
if(not HS) F3
if(not HS)send(P0,V)
send(P0,HS)
if(HS)G
if(not FS)M
if(not HS& not FS)
send(P0,ID)if(HS&
not FS)send(P1,ID)
171819
HS_INFS_IN
if(not HS)F1
if(FS)N
send(P0,FS)
if(not HS)F2
send(P0,HS)
if(HS)G
![Page 42: Real Time Systems Compilation with Lopht Dumitru Potop-Butucaru INRIA Paris/ AOSTE team Jan. 2016 1](https://reader035.vdocuments.site/reader035/viewer/2022062504/5a4d1b6d7f8b9ab0599b3f52/html5/thumbnails/42.jpg)
But this is not enough (1/3)
• Example: software pipelining
42
012345
P0 P1 P2time
6789101112
BusHS_IN
Generated schedule:- Latency: 17 - Throughput:
1/13
Software pipelining (classical compil. technique) can be applied.
FS_IN
if(not HS)F1
if(not HS)F2
if(FS)N
send(P0,FS)if(not HS) F3
if(not HS)send(P0,V)
send(P0,HS)
if(HS)G
if(not FS)M
if(not HS& not FS)
send(P0,ID)if(HS&
not FS)send(P1,ID)
Using software pipelining required some modifications to classical algorithms:• Bi-criteria: latency first, throughput second• Improved predication handling (conditions between cycles, no delay slot needed)
![Page 43: Real Time Systems Compilation with Lopht Dumitru Potop-Butucaru INRIA Paris/ AOSTE team Jan. 2016 1](https://reader035.vdocuments.site/reader035/viewer/2022062504/5a4d1b6d7f8b9ab0599b3f52/html5/thumbnails/43.jpg)
43
But this is not enough (1/3)
• Application
MC
Init = 1
if(m=3)F3
if(m=2)F2
if(m=1)F1
G
m=1 m=2 m=3
![Page 44: Real Time Systems Compilation with Lopht Dumitru Potop-Butucaru INRIA Paris/ AOSTE team Jan. 2016 1](https://reader035.vdocuments.site/reader035/viewer/2022062504/5a4d1b6d7f8b9ab0599b3f52/html5/thumbnails/44.jpg)
44
But this is not enough (1/3)
• Latency-optimizing scheduling
MC@true
F1@m=1
F2@m=2
F3@m=3
G@m=1
G@m=2
G@m=3
temps P1 P2
0
1
2
3
4
5
6
Latency = 7Throughput = 1/7
![Page 45: Real Time Systems Compilation with Lopht Dumitru Potop-Butucaru INRIA Paris/ AOSTE team Jan. 2016 1](https://reader035.vdocuments.site/reader035/viewer/2022062504/5a4d1b6d7f8b9ab0599b3f52/html5/thumbnails/45.jpg)
45
But this is not enough (1/3)
• Pipelined scheduling
– Using mode transition information
MC@true
F1@m=1
F2@m=2
F3@m=3
G@m=1
G@m=2
G@m=3
temps P1 P2
0
1
2
3
Latency = 7Throughput = 1/4
G@m=2
G@m=3
![Page 46: Real Time Systems Compilation with Lopht Dumitru Potop-Butucaru INRIA Paris/ AOSTE team Jan. 2016 1](https://reader035.vdocuments.site/reader035/viewer/2022062504/5a4d1b6d7f8b9ab0599b3f52/html5/thumbnails/46.jpg)
46
But this is not enough (1/3)
• Pipelined scheduling
– Without using mode transition information
MC@true
F1@m=1
F2@m=2
F3@m=3
G@m=1
G@m=2
G@m=3
temps P1 P2
0
1
2
3
4
Latency = 7Throughput = 1/5
G@m=1
G@m=2
![Page 47: Real Time Systems Compilation with Lopht Dumitru Potop-Butucaru INRIA Paris/ AOSTE team Jan. 2016 1](https://reader035.vdocuments.site/reader035/viewer/2022062504/5a4d1b6d7f8b9ab0599b3f52/html5/thumbnails/47.jpg)
47
But this is not enough (1/3)
• Software pipelining => memory replication– Rotating registers– Conditional execution => not clear which version of a
register to access• Need an indirection table, dynamically updated
• Real-time guarantees– Worst-case analysis– Account for artefacts• E.g. rotating register operations
![Page 48: Real Time Systems Compilation with Lopht Dumitru Potop-Butucaru INRIA Paris/ AOSTE team Jan. 2016 1](https://reader035.vdocuments.site/reader035/viewer/2022062504/5a4d1b6d7f8b9ab0599b3f52/html5/thumbnails/48.jpg)
48
But this is not enough (2/3)• Time triggered applications
– Target: ARINC 653-compliant OSs, time-triggered comm. • Synthesis of inter-partition communication code (and considering it during
scheduling)– Non-functional properties taken as input:
• ARINC 653 partitioning (all or part or none, application and platform)• Real time (release dates, deadlines – can represent end-to-end latency)• Preemptability of each task• Allocation requirements
– Optimizations:• Deadline-driven scheduling (to improve schedulability)• Minimize the number of tasks at code generation (at most one per input
acquisition point)• Minimize the number of partition changes
– Hypotheses: The scheduler and I/O impact on WCETs can be bounded.
![Page 49: Real Time Systems Compilation with Lopht Dumitru Potop-Butucaru INRIA Paris/ AOSTE team Jan. 2016 1](https://reader035.vdocuments.site/reader035/viewer/2022062504/5a4d1b6d7f8b9ab0599b3f52/html5/thumbnails/49.jpg)
49
But this is not enough (3/3)
• Many-core (bare metal)– Sound architecture abstraction layer with support for efficiency
• Precise platform modeling:– NoC: one resource per DMA and NoC arbiter
» Wormhole scheduling synchronizes schedules along the path of each packet– Memory banks are resources
» Reserve them for data, code, and stack to avoid contentions• WCET analysis for parallel code
– Considers synchronization overheads
– Synthesis of communication and synchronization code (lock-based) on the processors• Account for communication/synchronization initiation costs
– Precomputed preemptions for communications• Account for preemption costs
![Page 50: Real Time Systems Compilation with Lopht Dumitru Potop-Butucaru INRIA Paris/ AOSTE team Jan. 2016 1](https://reader035.vdocuments.site/reader035/viewer/2022062504/5a4d1b6d7f8b9ab0599b3f52/html5/thumbnails/50.jpg)
50
But this is not enough (3/3)
• Synchronized & preemptive comm.
xy
zu
f
g
Tile(1,1)
DMA(1,1)
N(1,1)(1,2)
N(1,2)(2,2)
In(2,2)
Tile(2,2)
f z
y y
u
hxxx xx
x xx
500
1000
1500
2000
2500
0
g
x
x x x
![Page 51: Real Time Systems Compilation with Lopht Dumitru Potop-Butucaru INRIA Paris/ AOSTE team Jan. 2016 1](https://reader035.vdocuments.site/reader035/viewer/2022062504/5a4d1b6d7f8b9ab0599b3f52/html5/thumbnails/51.jpg)
51
Other related results (1/3)
• Many-core platform with support for efficient predictability– Principle: avoiding contentions through mapping
• No shared caches • Hardware locks for mutual exclusion during RAM access
– Expose NoC arbitration to programming• Efficiency bottleneck is in both CPUs and interconnect, and similar
arbitration is required of both (in various contexts), so they should provide the same level of control.
• General-purpose: – No change in processor programming (gcc)– Not programming the NoC arbiters maintains functional correctness
• Small hardware overhead• Automatic mapping (global optimization) -> good results !
![Page 52: Real Time Systems Compilation with Lopht Dumitru Potop-Butucaru INRIA Paris/ AOSTE team Jan. 2016 1](https://reader035.vdocuments.site/reader035/viewer/2022062504/5a4d1b6d7f8b9ab0599b3f52/html5/thumbnails/52.jpg)
52
Other related results (2/3)
• Heuristics vs. exact methods (SAT/SMT/etc.)– Recent constraint solvers (Cplex, Gurobi, Yices2, etc.)
have largely improved performance – Question: Can they solve scheduling problems ?– Answer: Yes, up to a certain size, which largely
depends on problem complexity, problem type (schedulability vs. optimization), system load.• From a few tasks (optimization, preemptive, multi-periodic)
to more than 100 tasks.– Conclusion: Developing heuristic methods still has its
use, especially for complex problems like ours.
![Page 53: Real Time Systems Compilation with Lopht Dumitru Potop-Butucaru INRIA Paris/ AOSTE team Jan. 2016 1](https://reader035.vdocuments.site/reader035/viewer/2022062504/5a4d1b6d7f8b9ab0599b3f52/html5/thumbnails/53.jpg)
53
Other related results (3/3)
• Automatic delay-insensitive implementation of synchronous programs– Objective: separation of concerns between functional and
non-functional aspects• Optimal synchronization protocols ensuring functional determinism
– Characterization of delay-insensitive synchronous components (weak endochrony)• Set theoretical limits
– Weak endochrony checking• New representation of the behavior of concurrent systems by
means of sets of atomic behaviors– Synthesis of delay-insensitive concurrent implementations
![Page 54: Real Time Systems Compilation with Lopht Dumitru Potop-Butucaru INRIA Paris/ AOSTE team Jan. 2016 1](https://reader035.vdocuments.site/reader035/viewer/2022062504/5a4d1b6d7f8b9ab0599b3f52/html5/thumbnails/54.jpg)
54
Avionics GNC application
• Simplified version:– 3 tasks TFast ,TGNC et Tthermal
– 3 partitions PFast, PGNC et Pthermal
– WCETs: 40ms, 200ms et 100ms– Periods: 100ms, 1s, 1s
Fast GNC Thermal
![Page 55: Real Time Systems Compilation with Lopht Dumitru Potop-Butucaru INRIA Paris/ AOSTE team Jan. 2016 1](https://reader035.vdocuments.site/reader035/viewer/2022062504/5a4d1b6d7f8b9ab0599b3f52/html5/thumbnails/55.jpg)
55
Avionics GNC application
• Simplified version: – 3 tasks TFast ,TGNC et Tthermal
– 3 partitions PFast, PGNC et Pthermal
– WCETs: 40ms, 200ms et 100ms– Periods: 100ms, 1s, 1s– Hyperperiod expansion : MTF = 1s
Fast Fast FastFastFastFast Fast Fast FastFast
GNCThermal
D
D
![Page 56: Real Time Systems Compilation with Lopht Dumitru Potop-Butucaru INRIA Paris/ AOSTE team Jan. 2016 1](https://reader035.vdocuments.site/reader035/viewer/2022062504/5a4d1b6d7f8b9ab0599b3f52/html5/thumbnails/56.jpg)
56
Avionics GNC application
• Real time requirements– Tfast reads inputs arriving periodically in an infinite
buffer
Fast Fast FastFastFastFast Fast Fast FastFast
GNCThermal
D
D
0 100 200 300 400 500 600 700 800 900 1000
![Page 57: Real Time Systems Compilation with Lopht Dumitru Potop-Butucaru INRIA Paris/ AOSTE team Jan. 2016 1](https://reader035.vdocuments.site/reader035/viewer/2022062504/5a4d1b6d7f8b9ab0599b3f52/html5/thumbnails/57.jpg)
57
Avionics GNC application
• Real time requirements– Tfast reads inputs arriving periodically in a buffer of
size 2
Fast Fast FastFastFastFast Fast Fast FastFast
GNCThermal
D
D
0 100 200 300 400 500 600 700 800 900 1000
![Page 58: Real Time Systems Compilation with Lopht Dumitru Potop-Butucaru INRIA Paris/ AOSTE team Jan. 2016 1](https://reader035.vdocuments.site/reader035/viewer/2022062504/5a4d1b6d7f8b9ab0599b3f52/html5/thumbnails/58.jpg)
58
Avionics GNC application
• Real time requirements– Infinite buffers– « A full flow formed of 10 instances of Tfast
followed by an instance of TGNC and then by an instance of Tfast should take less than 1400 ms »
Fast Fast FastFastFastFast Fast Fast FastFast
GNCThermal
D
D
0 100 200 300 400 500 600 700 800 900 1000
![Page 59: Real Time Systems Compilation with Lopht Dumitru Potop-Butucaru INRIA Paris/ AOSTE team Jan. 2016 1](https://reader035.vdocuments.site/reader035/viewer/2022062504/5a4d1b6d7f8b9ab0599b3f52/html5/thumbnails/59.jpg)
59
Avionics GNC application
• Step 1 : remove delayed dependencies– Replace them by timing barriers– Heuristic approach (sub-optimal)
Fast Fast FastFastFastFast Fast Fast FastFast
GNCThermal
D
D
0 100 200 300 400 500 600 700 800 900 1000
![Page 60: Real Time Systems Compilation with Lopht Dumitru Potop-Butucaru INRIA Paris/ AOSTE team Jan. 2016 1](https://reader035.vdocuments.site/reader035/viewer/2022062504/5a4d1b6d7f8b9ab0599b3f52/html5/thumbnails/60.jpg)
60
Avionics GNC application
• Step 1 : remove delayed dependencies– Replace them by timing barriers– Heuristic approach (sub-optimal)
Fast Fast FastFastFastFast Fast Fast FastFast
GNCThermal
0 100 200 300 400 500 600 700 800 900 1000
1300
![Page 61: Real Time Systems Compilation with Lopht Dumitru Potop-Butucaru INRIA Paris/ AOSTE team Jan. 2016 1](https://reader035.vdocuments.site/reader035/viewer/2022062504/5a4d1b6d7f8b9ab0599b3f52/html5/thumbnails/61.jpg)
61
Avionics GNC application
• Step 2 : off-line scheduling– Allocate and schedule blocks one by one (no backtracking)– Deadline-driven
• Of all tasks that can be executed, consider the one with earliest deadline
• Deadline = minimum of all successor deadlines
Fast Fast FastFastFastFast Fast Fast FastFast
GNCThermal
0 100 200 300 400 500 600 700 800 900 1000
1300
![Page 62: Real Time Systems Compilation with Lopht Dumitru Potop-Butucaru INRIA Paris/ AOSTE team Jan. 2016 1](https://reader035.vdocuments.site/reader035/viewer/2022062504/5a4d1b6d7f8b9ab0599b3f52/html5/thumbnails/62.jpg)
62
Avionics GNC application
0 100 200 300 400 500 600 700 800 900 1000
GNC is pipelined
Fast Fast FastFastFastFast Fast Fast FastFast
GNC
100 200 300 400 500 600 700 800 900 1000
Thermal
1300
0
• Step 2: off-line scheduling
0 ∞
![Page 63: Real Time Systems Compilation with Lopht Dumitru Potop-Butucaru INRIA Paris/ AOSTE team Jan. 2016 1](https://reader035.vdocuments.site/reader035/viewer/2022062504/5a4d1b6d7f8b9ab0599b3f52/html5/thumbnails/63.jpg)
63
Avionics GNC application
0 100 200 300 400 500 600 700 800 900 1000
• Step 3: post-scheduling optimization
– Deadline-driven scheduling routine• Many context/partition changes• Reduce their number by moving reservations subject to timing and data dependency requirements
– Low complexity
– Possible results
0 100 200 300 400 500 600 700 800 900 1000Infinitebuffer
0 100 200 300 400 500 600 700 800 900 1000
Buffer of size 3
![Page 64: Real Time Systems Compilation with Lopht Dumitru Potop-Butucaru INRIA Paris/ AOSTE team Jan. 2016 1](https://reader035.vdocuments.site/reader035/viewer/2022062504/5a4d1b6d7f8b9ab0599b3f52/html5/thumbnails/64.jpg)
Avionics GNC application
• Full application– 4 processors, 1 broadcast bus– 13 tasks before hyper-period expansion– 7 partitions– 10 end-to-end latency constraints– Result:• Processor loads: 82%, 72%, 72%, 10% (last one for
telemetry)• Bus: 81%
![Page 65: Real Time Systems Compilation with Lopht Dumitru Potop-Butucaru INRIA Paris/ AOSTE team Jan. 2016 1](https://reader035.vdocuments.site/reader035/viewer/2022062504/5a4d1b6d7f8b9ab0599b3f52/html5/thumbnails/65.jpg)
65
Conclusion (1/3)
• Full-fledged real-time systems compilation is feasible – At least for certain application classes– Ensures correctness and efficiency– Allows a trial-and-error design approach for complex
embedded systems
• Simple theoretical and algorithmic principles– Independence, isolation, list scheduling, timetables,
pipelining…
![Page 66: Real Time Systems Compilation with Lopht Dumitru Potop-Butucaru INRIA Paris/ AOSTE team Jan. 2016 1](https://reader035.vdocuments.site/reader035/viewer/2022062504/5a4d1b6d7f8b9ab0599b3f52/html5/thumbnails/66.jpg)
66
Conclusion (2/3)
• But no not mistake a list of simple principles for a compiler– Integration requires careful adaptation of the concepts
and techniques– Match the needs of realistic applications to give good
results• Good resource use• Good timing precision
– Few percent difference between guarantees and actual execution for certain many-core applications.
– Lopht: not a scheduling toolbox, nor a classical compiler• Hence the title « real time systems compiler »
![Page 67: Real Time Systems Compilation with Lopht Dumitru Potop-Butucaru INRIA Paris/ AOSTE team Jan. 2016 1](https://reader035.vdocuments.site/reader035/viewer/2022062504/5a4d1b6d7f8b9ab0599b3f52/html5/thumbnails/67.jpg)
67
Conclusion (3/3)
• Major difficulties– Few benchmarks/case studies• Sharing is difficult (technical, legal, will issues)
– Publication• Cross-domain, everybody sees it as interesting « for the
other domain » (sort of NIMBY syndrome)– Sharing between various targets• No general platform definition language
![Page 68: Real Time Systems Compilation with Lopht Dumitru Potop-Butucaru INRIA Paris/ AOSTE team Jan. 2016 1](https://reader035.vdocuments.site/reader035/viewer/2022062504/5a4d1b6d7f8b9ab0599b3f52/html5/thumbnails/68.jpg)
68
Future work (1/2)
• Short/medium term (ongoing):– Execution platform modeling– Provide compiler correctness guarantees/proofs– Improve treatment of multi-periodic applications
• Trade-off between scheduling freedom (and thus efficient resource use) and compactness of representation (architectural limits, legibility)
– New platforms• TTEthernet – taking into account platform-related tools• Kalray MPPA – Generate code over the Kalray hypervisor, NoC
handling– New case studies for/from the industry
![Page 69: Real Time Systems Compilation with Lopht Dumitru Potop-Butucaru INRIA Paris/ AOSTE team Jan. 2016 1](https://reader035.vdocuments.site/reader035/viewer/2022062504/5a4d1b6d7f8b9ab0599b3f52/html5/thumbnails/69.jpg)
69
Future work (2/2)
• Longer term: – Improve embedded systems mapping by
adapting/combining models and techniques from compilation, real-time scheduling, and synchronous languages• Compilation: Code replication, Loop unrolling…• Real-time scheduling: CRPD computation…• Synchronous languages: n-synchronous clock calculus…
![Page 70: Real Time Systems Compilation with Lopht Dumitru Potop-Butucaru INRIA Paris/ AOSTE team Jan. 2016 1](https://reader035.vdocuments.site/reader035/viewer/2022062504/5a4d1b6d7f8b9ab0599b3f52/html5/thumbnails/70.jpg)
70