track a-shmuel panijel, windriver

Developing Embedded Developing Embedded

Systems with Multicore Systems with Multicore

and Virtualizationand VirtualizationSamuel PanijelSamuel Panijel

May 4, 2011 1

Samuel Panijel

Regional Engineering Manager

Wind River Q2-2011

Samuel Panijel

Regional Engineering Manager

Wind River Q2-2011

Context• Multicore and virtualization are creating a disruption

in the embedded industry

– Multicore provides more processing power for less power

– Virtualization provides partitioning on single and multicore

silicon

• Wind River provides a broad portfolio of technology

to assist the migration to multicore

May 4, 2011 2| © 2010 Wind River. All Rights Reserved.2

to assist the migration to multicore

– Runtimes, Middleware, Virtualization and Tooling

• This presentation will focus on the use of

virtualization

– For single and multicore

– Based on the Wind River Hypervisor for Embedded

Agenda

• Multicore and virtualization overview

– An introduction to AMP, SMP and Virtualization

• Virtualization capabilities

• Virtualization details


– Modes of virtualization

– The mechanisms behind virtualization

– Virtualizing memory, cores, devices, interrupts

– Inter-board communication

How to design your next generation of devices

Primary Multicore Software

Configurations

Single CoreCPU

OS

“Traditional” Virtualization

CPU

OS OS

Hypervisor


Multicore

OS: Could be VxWorks, Wind River Linux, or other executive or OS

Arbitrary combinations of these primary configurations can be used to create more

advanced configurations.

OS

Core 2Core 1

OS

Supervisor

Supervised AMP (sAMP)SMP

OS

Core 1 Core 2

OS

Core 2Core 1

OS

Unsupervised AMP

Symmetric Multiprocessing (SMP)

• A lot of experience

• Works well for parallel algorithms

• OS performs scheduling and load balancing

• A viable solution, depending on your needs


• A viable solution, depending on your needs

– Does not always scale with many cores

• Linux and VxWorks SMP

SMP

OS

Core 1 Core 2

Asymmetric Multiprocessing (AMP)

• One operating system per core

• Questions raised in regard to configuring and

run-time

– Who “owns” devices?

– How is memory split?

May 4, 2011 6

| © 2010 Wind River. All Rights Reserved.6

– How is memory split?

– How is memory protected?

– How is this configured?

– How is this built into the kernel?

• Mixed operating systems?

OS

Core 2Core 1

OS

Unsupervised AMP

Choosing Supervised AMP

What Is sAMP?

– AMP, but guarded by a supervisor

When to Use sAMP

– Memory and device partitioning (protection, robustness)


– Flexibility in the mapping of hardware to software

– Memory virtualization (each OS starts at 0x0)

– Reuse of the same image multiple times

– Out-of-the-box life cycle management

OS

Core 2Core 1

OS

Supervisor

Supervised AMP (sAMP)

Agenda






May 4, 2011 10







Virtualization

• Partitioning is a benefit of virtualization

• Virtualization partitions the hardware into virtual boards

• The virtualization layer gives the board access to

– Virtualized processing cores (optional)

– Virtualized memory

– Virtualized devices and interrupts

May 4, 2011 11


– Virtualized devices and interrupts

• Virtualization can be done in a scalable fashion

| © 2009 Wind River. All Rights Reserved.

Core Virtualization Guest Protection

Resource Partitioning Boot Life Cycle Management Debugging

Core Protection InterruptsVirtual Memory and Devices

Hypervisor

Protected Supervisor

Supervisor

An Example Virtualization Scenario

Virtual Board 3Virtual Board 1

VCore Memory Timer

Ethernet

VxWorks

Application

Virtual Board 2

VCore Memory Timer

IDE Serial

Wind River Linux

Application

VCore Memory Timer

Application

Ethernet

MIPC

May 4, 2011 12


VCore Memory Timer VCore Memory Timer VCore Memory Timer

Wind River Hypervisor

Physical Board

IDE Ethernet Memory CoreTimer Serial Core

Configuration

Summary of the Example

• Virtual boards

– Virtual board scheduling if required

• Virtualized memory

• Virtualized processing cores

• Virtualized devices


• Virtualized devices

• Inter-board communication

• Configuration and debuggingVirtualization provides new and flexible design capabilities

for single and multicore systems

Virtualization Performance

• Virtualization needs to have minimal cost

• Wind River and customer benchmarking

– sAMP: Overall cost is less than 1%

– Core virtualized: With significant load only a few percent,

similar to running multiple processes

– Some additional latency to interrupt handling

May 4, 2011 14


– Some additional latency to interrupt handling

• Actual performance depends on

– Processor, board, application

– Multicore configuration

• The additional design capabilities and flexibility

typically outweigh any potential overhead

Agenda












Virtualization Principles

The hypervisor adds an additional layer of indirection

to between the OS and the physical hardware

The hypervisor virtualizes the hardware and executes

guest operating systems. The level of virtualization can

be configured based on device needs


be configured based on device needs

Hypervisor

Guest Guest Guest

Hypervisor Mechanics

• OS privilege modes

• Emulation vs. paravirtualization vs. hardware

assist

• Different types of hypervisors


• Memory

• Devices, interrupts

• Inter board communication

Privilege Modes

Operating systems work in privilege modes:

� The OS kernel itself can do anything with the

hardware

� Device drivers are more restricted

� User applications/components are least privileged


Intel privilege rings source: Wikipedia

Ring 3

Ring 2

Ring 1

Ring 0

Kernel

Device Drivers

Device Drivers

Applications

Least Privileged

Most Privileged

Example based on Intel privilege rings, other processors have similar

concepts

Hypervisor Is The Most Powerful

Reserve the highest privilege mode for the hypervisor.

Lower operating system privilege modes through

– Emulation

– ParavirtualizationRing 3

Ring 2 Least Privileged


– Paravirtualization

– Hardware assist

Hypervisor

Ring 1

Ring 0

Kernel

Device Drivers

Device Drivers

Applications

Most Privileged

Approaches to Virtualization

Type 2 – Hosted

Hardware Emulation

System Virtualization

Hardware Virtualization

Type 1 – Native

May 4, 2011 21


Paravirtualized Full Virtualization

(Transparent)

Dynamic Binary

Translation

Hardware

Assisted

Privilege Modes – Hardware Assist• Privilege modes are processor specific. You cannot just “add”

another mode. Only the silicon vendor can:

– Freescale e500mc

– Intel VTx

– Virtualization-aware instruction set (aka hardware-assisted

virtualization)

• If not available, paravirtualization will be more for the OS


• If not available, paravirtualization will be more for the OS

developer.

Often the truth is in the middle: a mix of software and

hardware assist.

Privilege Modes – Paravirtualization

• Modify the guest operating system, to collaborate

with the hypervisor.

• Reduce the privilege level during design time.

• Provide a collaborative system approach.

• Use for embedded.


• Use for embedded.

• Handle devices:

– In the hypervisor (impact to hypervisor size and stability)

– In a specially crafted guest (guest running in privilege)

– Directly assigned to guest

Mixing Technologies

Mixing of Paravirtualization and Hardware Assist

• Use the best of both worlds for embedded

– Paravirtualization collaboration of OS with

hypervisor (avoid privilege code and call

hypervisor instead)

May 4, 2011 24

hypervisor instead)

– Hardware assist using silicon (great for address

translation)

Wind River hypervisor uses a mix of paravirtualization

and hardware assist, if the processor supports it.

24

| © 2010 Wind River. All Rights Reserved.

Type 1 vs. Type 2 Hypervisor

A type 1 hypervisor runs directly on the hardware,

managing resources, and providing services to guests

A type 2 hypervisor is hosted in another operating

system (the host)


Guest

Hypervisor

Type 1

Host

Hypervisor

Guest

Type 2

The difference between type 1 and type 2 is not as simple and useful as it seems

Embedded vs. IT Hypervisors

Guest

Guest

Hypervisor

Guest

Guest GuestDom0

Driver

Emu Driver

Driver

Driver EmuDriver


Slim and Fat provide more IT-level capabilities (VM migration, load balancing, full

device virtualization) and hence carry an overhead cost. Wind River Hypervisor is

custom built for embedded and provides better performance

Embedded

Hypervisor

IT – Slim

Hypervisor

IT – Fat

Hypervisor

Driver

Manage Memory

• Every guest expects memory from 0x0 – MAXMEM.

• Hypervisor provides translation and protection.

• It uses flat or guest virtual memory.


Guest

0x0 MAXMEM

0x0 MAXPHYSMEM

Guest Guest

Virtual

Physical

• Possible to provide noncontinuous mapping

• Possible to provide shared memory

– Shared segment can be put anywhere in guest memory

Manage Memory – Special Mappings


0x0

Physical

GuestGuest Guest

Virtual

Shared

Device SupportDriver

Direct

� One owner

� Others shareShared

Guest

Driver

Guest Guest

shDev Driver

� Fastest, native performance

� Recommended where possible

May 4, 2011 34


HypervisorDriver� Driver in hypervisor

� Stubs in the guestVirtualized

Guest

Stub

Hypervisor

Guest

Stub

Driver

Guest

Driver1

Hypervisor

Guest

Driver1

Driver1 Emulation

Driver2

� Driver in hypervisor

� Emulation in hypervisor

� Regular driver in guest

Emulated

Devices Access – Virtual Interrupts

Virtual Board

Virtual Interrupt Controller

Guest

Virtual Board

Pending

Interrupt

Guest

Virtual Interrupt Controller

Running

May 4, 2011 38



Pending Interrupt

Pending Interrupt

Interrupt Controller

Core Virtualization – Schedulers

• Two different schedulers available

• Priority

– Each virtual board has a priority

– Highest priority board that is non-idle is on-processor

– Boards with the same priority are scheduled round-robin

• Time slice

May 4, 2011 40


• Time slice

– Each virtual board gets a designated time slice

– Only single core

• Scheduler is pluggable

– Other scheduling algorithms can be plugged into the

hypervisor

Inter-Board Communication

• Shared memory can be given to each board

– Easy way to share large amounts of data

– Synchronization through asynchronous messaging

• For signaling, or using zero-copy data sending

• MultiOS Inter Processor Communication (MIPC)

• Asynchronous messaging promotes parallel algorithms

May 4, 2011 42


• MultiOS Inter Processor Communication (MIPC)

– Fast communication for multicore and virtualization

– Socket-like API for easy migration

– Wind River standard API between OSes in any

configuration

– Uses shared memory and interrupts

Services on top of MIPC

• MIPC Network Device (MND)

– TCP/IP layered on top of MIPC

– Slower than MIPC, but offers TCP/IP layer

• MIPC Serial Device (MSD)

– Ability to use a single serial port with multiple


– Ability to use a single serial port with multiple

guests

– For terminal access, as well as printf for example

for debugging

• MIPC File System (MFS)

The How – Tailored IPCs

• Shared memory

– High performance

• MIPC

– High performance, zero copy

– OS independent

– 20K footprint

– Hardware support

Physical Board

Physical Board

Core 2

Virtual Board

App 6

MIPC

Core 1

Virtual Board

App 5

MIPC

Virtual Board

App 4

Linux 2

MIPC

Linux 1VxWorks

May 4, 2011 44


– Hardware support

– Static topology

• TIPC

– 100K footprint

– Full-featured

– Transparent multiboard

– Dynamic topology

Physical Board

Physical

Board

Core

Virtual Board

TIPC

Ethernet

App

Ethernet

App

TIPC

Virtual Board

TIPC

App

Linux 1Linux 3Linux 2

Building a System (Standard

Approach)

Hypervisor ELFImage (hypervisor.elf)

VxWorks ELFImage (vxworks.elf)

Linux ELF

Boot Image(ELF format)

Linux

Packaging

May 4, 2011 45

Linux ELFImage (vmlinux.elf)

XML Configuration

Image Manifest

VxWorks

Hypervisor

PackagingTool

Booting and Lifecycle (Standard Approach)

VB1

VxWorks

App


Boot Image(ELF format)

Image Manifest

VxWorks

Linux

Hypervisor

VB2

Linux

App

May 4, 2011 46

• Hypervisor boots from ROM file system

• Hypervisor first (very quickly), then the appropriate VB

• VBs can be stopped, reloaded and restarted from initial image

• VB could contain a bootloader

46

Hypervisor

Virtualization Details – Summary

• Modes of virtualization

• The mechanisms behind virtualization

• Virtualizing memory, cores, devices, interrupts

• Inter-board communication


Ring 3

Ring 2

Ring 1

Ring 0

Kernel

Device Drivers

Device Drivers

Applications

Ring 3

Ring 2

Ring 1

Ring 0

Kernel

Device Drivers

Device Drivers

Applications

Ring 3

Ring 2

Ring 1

Ring 0

Kernel

Device Drivers

Device Drivers

Applications

Hypervisor

Core 0 Core 1

Benefits of Virtualization

� Partition a system

� Evolve the partitioning

� Protect partitions

� Reduced time to market

� Reduced risk

� Reduced BOM

Provides the ability to … Resulting in …

May 4, 2011 49


� Protect partitions

� Reuse legacy code

� Share devices

� Verify a configuration

� Reduced BOM

� Investment protection

� Future proofing

Multi-Context Debugging

with OCD & Wind River Hypervisor

Debug Context ViewVisibility to

physical cores

- registers- bit level details- memory

- cacheTarget Manager

Multiple virtual boards (VBs)

running VxWorks in hypervised environment

May 4, 2011 54

environmentFine-grainedbreakpointcontrol withassociationto multiplecontexts

Deep visibility and granular control to debug complex system issues:Examples: inter-core race lock conditions, synchronization, message passing.

track a-shmuel panijel, windriver

Education

virtualization partitioning

interrupts virtualization

power virtualization

virtualization layer

multicore silicon wind

minimal cost wind river

core virtualized

core questions