evolution of mipi imaging conduit...with certificate signing • data signing -data is originating...

CONFIDENTIAL 1 | © 2019 MIPI Alliance, Inc.

Evolution of MIPI Imaging Conduit

October 18, 2019

Haran ThanigasalamMIPI Camera & Imaging WG Chair

2 | © 2019 MIPI Alliance, Inc.

Agenda • The Big Why

o Two key discoverieso Machine human frictiono Two pathways

• MIPI Imaging Conduit Solutiono Multi-phase developmento What’s Doneo What’s WIPo What’s Next

• Summary


The Big Why

URL

• Enrich lives by better understanding the world around us

• Knowledge limitations

• Scope of perception

• Challenges with abstraction

• Evolution of intelligence

• Two key discoveries

1. Individual & Collective Dysfunction

2. Superior Intelligence

URLURL

https://books.google.com/books/about/The_Mindful_Therapist_A_Clinician_s_Guid.html?id=gFo_zhxys0AC&printsec=frontcover&source=kp_read_button

https://news-media.stanford.edu/wp-content/uploads/2018/12/11155529/aigetty.jpg

https://www.theguardian.com/science/blog/2012/feb/28/how-many-neurons-human-brain


MIPI CSI-2 Imaging Conduit 2019

Machine human friction• Future is deeper human machine interaction

with connected devices

• Two Pathways:

1. Making us become more like machines

2. Making machines become more like us (machine awareness, emotional vision)

URLURL

https://scifi.stackexchange.com/questions/65398/why-does-data-have-to-use-a-computer-in-star-trek

https://memory-alpha.fandom.com/wiki/Star_Trek:_Borg


Evolving MIPI Imaging Solution

IngredientsPhoton Collectors, Transistors, Optics,

Emitters, Algorithms

ApplicationsBroad Spectrum of Imaging & Vision Use cases targeting

multiple platforms

CSI-2 Imaging Conduit

One World l One Imaging Conduit l MIPI CSI-2

URL

I. Mobile - Pristine photography & video streaming on mobile platforms [CSI-2 v1.x]- RES_FPS_BPP | PORT EXP | SNS SWITCHING

II. Platforms - Support broad range of imaging applications beyond photography on multiple platforms [CSI-2 v2.x, v3.x]- SCR | VCX | LRTE (PDQ, ALP) | USL | RAW-24 | DPCM | SROI

III. Awareness - Develop infrastructure for emotional intelligence and machine awareness for perception and decision making capabilities [CSI-2 v4.x]- AOSC | FSAF | ISEC |ADAPL

IV. Scale - Open System Cloud Imaging Applications utilizing vision analytics (Schools, Hospitals, Municipalities, States)

V. WIPComplex problems cannot be solved by an individual company

Need for global alliances and liaisons: MIPI | DMTF | IEEE 1722

Evolution of MIPI Imaging Conduit Infrastructure

https://www.google.de/search?q=artificial+intelligence+evolution&biw=1375&bih=776&site=webhp&source=lnms&tbm=isch&sa=X&sqi=2&ved=0ahUKEwjC3aqd_q7NAhXEaRQKHUZADu8Q_AUICCgB


Two different CSI architectures

M-PHYPHY

Transport

Applications

CSI-2UniPro

CSI-3

D-PHYC-PHYA-PHY

CSI-2

C-PHYD-PHYI3C A-PHY

• MIPI CWG is continuing to evolve and advance CSI-2 imaging conduit targeting multiple platforms:•I3C: Ultra-Low-Power AON•C/D-PHY: Performance •A-PHY: Long Reach

• CSI-3 is not a next-gen solution, nor superset, nor backwards compatible with CSI-2


What’s Done

SROI UNIFIED IMAGING DRIVER(CCS, DisCo*)

LRTE(PDQ, ALP)

USL(ENCAP, REPL, OPT_WIRE)

DPCM | RAW-24

PSD


PSD Reduction

CSI-2 over C-PHY PSD emission reduction with scrambling (embedded clock and data)

CSI-2 over D-PHY PSD emission reduction with scrambling (data lanes)


LRTE

• Dramatically Improve Sensor Aggregation

• Optimal Transport Preserving Integrity

• Real-time Perception & Decision Making

• Phase out EOS &IL Impediments

• Asymmetric SP/LP Spacers


Smart Region of Interest

https://fuse.wikichip.org/news/763/iedm-2017-sonys-3-layer-stacked-cmos-image-sensor-technology/

3-layer stacked image sensor

• Phase 2 Single Frame

• Phase 4 Multi Frame

• ARCH: Edge |Hybrid


DPCM Objective Qualification• Enable 10-bit compression of RAW-12 video image with better IQ than

prior version 12-8-12

• Reduce maximum absolute error of single-bit change in pixel value by a factor of 4.43x

• Qualified 5 degree slanted edge input image with low, medium, and high illumination levels:

– 12-10-12 virtually indistinguishable from original image

– MTF frequency response analysis closely tracks the original (HI/MI/LI - LC/MC/HC)

• Benefits include Link BW reduction cost savings


• Unified Imaging Software Driver

• Enable Image Sensor Capabilities for Vision Compute

• Accelerate Bring up on Reference Platforms

Algorithm is King

• Advance pathways to realize image sensor self-tuning


CSI-2 CCS


What’s WIP

ISEC• Develop provisions for end-to-end imaging conduit security

(interface and class)• Authentication - trustworthy / genuine sensor module

with certificate signing• Data signing - data is originating from trusted source

devoid of spoofing attack• Obfuscation - Data encryption at line rate

• External liaisons with Distributed Management Task Force Security Protocol & Data Model development• Authentication• Session key exchange

FSAFMission Critical Imaging Applications• End to End CRC Protection • Enhanced Frame Structure • Fault Injection Mode

AOSCVDSP

SNSMulti-Mode Image

APP

Point-To-PointCSI-2 over I3C

• Enable always-on surrounding awareness

• Infrastructure to accommodate rapid evolution of vision inferencing

• Ultra-low-power operations

A-PHY ADAPL• Enable native long-reach support for DISCRETE and

INTEGRATED image sensors


Always On Sentinel Conduit (AOSC)

VDSP

CSI-2 over C/D-PHYSNS

Dual-Mode Image Sensor(Cropped | Full-Frame)

APP

Application Processor

Point-To-PointCSI-2 over I3C

• Enable always-on surrounding awareness

• Infrastructure to accommodate rapid evolution of vision inferencing

• Ultra-low-power operations

Always On Imaging ApplicationsSNS Guidance

<2mw @ 3FPS, 8-10BPP, QVGA Monochrome

Select applications may require higher capabilities


End-to-end Security • Develop provisions for end-to-end imaging

conduit security (interface and class)• Authentication - trustworthy / genuine

sensor module with certificate signing• Data signing - data is originating from

trusted source devoid of spoofing attack• Obfuscation - Data encryption at line rate

• External liaisons with Distributed Management Task Force Security Protocol & Data Model development• Authentication• Session key exchange


CSI-2 ISEC Considerations Device Manufacturing A device shall not reveal the device unique asymmetric private key to device manufacturing personnel during manufacturing.

Device shall only reveal the device unique asymmetric private key to authorized device manufacturing personnel while programming the device.

Platform Manufacturing A device shall not reveal the device unique asymmetric private key or a host shall not reveal the host unique asymmetric private key to platformmanufacturing personnel.A device shall only reveal device unique symmetric key to authorized platform manufacturing personnel while programming the device on the CSI-2 linkinterface.

Platform Refurbishing A device shall not reveal the device unique asymmetric private key to platform refurbishing personnel.

A device shall only reveal device unique symmetric key to authorized platform refurbish personnel while programming the device on the CSI-2 link interface.

Host Firmware Update The host firmware update shall not require a retransmit of the device unique symmetric key on the CSI-2 link.

Model Authentication Host shall be able to determine certified device model during platform manufacturing / refurbishing. The certification level shall be available throughmetadata published by the device vendor.

Device Authentication Host shall be able to determine that the device has not changed.Device Session Key Rotation Device shall support rotation of session keys for runtime activities.Device Firmware Measurement Device shall generate the mutable firmware measurement response using only its immutable code or HW

Device Firmware Measurement Replay Host shall be able to determine that device firmware measurement is replay protected.Device Firmware Rollback Host shall provide a policy to prevent device firmware rollback.Device Data Integrity Host shall be able to determine that received device data is the same as data captured from the image sensors on the device without modification or replay

of the previously captured data.Device Data Confidentiality Device may protect Privacy sensitive data from a rogue actor able to observe the CSI-2 link. For long reach IOT platforms Host/Device shall protect Privacy

sensitive data from a rogue actor able to observe the long reach cable.

Host Authentication Device shall be able to determine the identity of the host in a single -master environment. Device may be able to determine the identity of the host in amulti-master environment.

Host Command Integrity Device shall be able to determine the integrity of the host command including region of interest, virtual channel etc.

Device Revocation Host shall be able to revoke a vulnerable device.Device Tamper Detection Device shall indicate tamper evidence in its response.


CSI-2 Functional Safety

Mission Critical Imaging Applications (ISO-262262: 2018)• End to End CRC Protection • Enhanced Frame Structure • Fault Injection Mode


CSI-2 v4.0 Imaging Conduit

• Always On Sentinel Controller (AOSC)– Optimal infrastructure for VDSP: ultra-low power, simple, always on CSI-

2 transport mechanism using I3C I/O – Target Platforms: Mobile, IoT & Client, Autonomous

• Functional Safety (FSAF)– Develop safety provisions for mission critical decision making

applications on autonomous platforms with areas of interest may include Message Counter (MC), SourceID, End-To-End CRC/MC protection, Enumeration of Rows & Columns.

– The solution shall also meet the needs of Automotive ISO-26262: 2018 requirements for ASIL-B to ASIL-D.

– Target Platforms: IoT, Autonomous (ADAS)

• Imaging Security (ISEC)– Develop provisions for end-to-end imaging conduit authentication &

data protection required for privacy.– Establish liaisons with DMTF – Target Platforms: Mobile, IoT & Client, Autonomous

• Adaptation Layer (ADAPL) for A-PHY– Support integrated and discrete solutions couple to CSI-2 over C/D-PHY– Long-reach SerDes solution for imaging applications.– Target Platforms: Autonomous (ADAS)

• Time Synchronization Provision (TSP)• Explore development of time synchronization including 1)

Transport and 2) Enablement• Target Platforms: Mobile, IoT & Client, Autonomous

• Multi-Sensor Support (MSS) • Explore supporting two or more C/D-PHY image sensors

coupled to single RX port on Application Processor• Target Platforms: Mobile, IoT (AR / Wearable Headgears),

Client, Autonomous

• Radar and LiDar Enhanced Support• Explore enhanced provisions to transport RAW data

from Radar and LiDAR image sensors for ADAS

• Unified Imaging SW Driver | Self Tuning• Continue to advance CCS aligned with CSI-2

• Develop DisCo for Imaging v1.0 with SWWG

• Alleviate Product Interoperability Risk• Continue to advance ETE CTS for CSI-2 (X-Matrix)

HW & System HW & System

SW & Drivers

Interoperability

PHYs & I/Os: A/C/D-PHY & I3C


Problem Definition (Use Cases)| Timing (Execution Vs Perfection)

Innovation Growth Maturity

Phase I(Mobile)

Phase II(Platforms)

Phase III(Awareness)

MIPI CWG

Time

Phase IV(Scale)


PHYs & I/Os

CSI-2

IEEE-1722

IEEE 802.3(ETH)

IEEE 802.11(WiFi)

CSI-2

C-PHYD-PHYI3C

A-PHY (DIS)

CSI-2

C-PHYD-PHYI3C A-PHY(INT)

Natively Supported• I3C: Ultra-Low-Power• D-PHY: Performance• C-PHY: Performance • A-PHY: Long Reach

Potential Exploration / Evaluation• IEEE-1722 for ETH/WiFi


Closed System PHYs

• Closed System Mid-Range solutions targeting Q2 2019 development• Closed System Long-Range solution targeting Q1 2020 development• Potential IEEE 1722 evaluation for Open System developments


CSI-2 over C/D-PHYs (2 Lanes)

D-PHYRX

(6-pins)CSI-2

RX

D-PHYTX

(6-pins)CSI-2

TX

Periodic clock

Data

I2C Compatible 2-wire Camera Control

CSI-2 Imaging over D-PHY

Pin compatible

CSI-2D-PHY

ImageSensor

ApplicationProcessor

C-PHYRX

(6-pins)

C-PHYTX

(6-pins)

I2C Compatible 2-wire Camera Control

CSI-2 Imaging over C-PHY

CSI-2C-PHY

ImageSensor


Embedded clock & data

CSI-2TX

CSI-2RX

AF Gyro OIS MEMS

AF Gyro OIS MEMS

Flash

Flash

2.3x Effective Pixel BW

1x Effective Pixel BW


Unified Serial Link

D-PHYRX CSI-2

RX

D-PHYTXCSI-2

TX

Periodic FWD clock

Data0

Camera Control Interface (CCI) over I2C / I3C / SPI

ImageSensor Module


GPIOs

D-PHYTRX CSI-2

USL

D-PHYTRXCSI-2

USL

ImageSensor Module


C-PHYRX CSI-2

RX

C-PHYTXCSI-2

TX

EMB_CD_TRIO_0

Camera Control Interface (CCI) over I2C / I3C / SPI

ImageSensor Module


GPIOs

C-PHYTRX CSI-2

USL

C-PHYTRXCSI-2

USL

ImageSensor Module


CSI-2 USL over D-PHY CSI-2 USL over C-PHY

Periodic FWD clock

Data0

EMB_CD_TRIO_0

USL USL

• Builds on LRTE• Wire Reduction• Encapsulation• Replay Protection• Long reach


Latest CSI-2 v3.0 over C/D-PHY Port BW• CSI-2 v3.0 C/D-PHY BW capabilities per lane

– C-PHY v2.0: Standard Channel 6 GSps / Lane = 6 x 2.28 Gbps / 3 wires, Short Channel 8 GSps / Lane

– D-PHY v2.5: Standard Channel 4.5 Gbps / Lane = 4.5 Gbps using 4 wires, Short Channel 6.5 Gbps / Lane

– D-PHY v3.0 will provide higher BW– trending spec completion Q1 2020

CSI-2 v3.0 over C/D-PHY Imaging Conduit Effective (Usable) BW Solutions:

• 9 Wire CSI-2 over C-PHY: 41.1 Gbps [Standard Channel] (3 x 6 x 2.28)• 10 Wire CSI-2 over D-PHY: 18.0 Gbps [Standard Channel] (4 x 4.5)


CSI-2 over C-PHY - 2.3x Effective BW

• 3 terminated conductors

• CSI-2 policy: symbol transition each state for trivial CDR

• Theoretical BW gain: log2^5

• As implemented for CSI-2 = 16/7


CSI-2 over A-PHY• Discrete & Integrated• NRZ: Low Cost: NRZ (8b/10b)• PAM: Scalable Performance: PAM-X (i.e. PAM-4)• O: Optional | M: Mandatory• Cable Topologies (Image Sensor PCB à Processor ECU PCB)

– 50 Ohm Coax | 15 m | 4 inline connectors– 100 Ohm SDP | 10 m | 4 inline connectors

G1(2 Gbps)

G2(4 Gbps)

G3(8 Gbps)

G4(12 Gbps)

G5(16 Gbps)

NRZ

M M O

PAM

M M M

32/48/100 Gbps

PAM


Platform Considerations

APP

ISPCVE

AGG

SNS MODULE

IRXVCMGYROACCEPROMLEDLASER…

ARCH: Mobile | Client | IoT | Autonomous

CENTRAL

EDGE / DISTRIBUTED

HYBRID

PHY

DISCRETE

INTEGRATED

GFX

SNS

VDSP

AOSC

CSI-2 v4.0 FSAF DIS

CSI-2 v4.0 FSAF EN

CSI-2 v4.0 FSAF ENUSL ENLRTE EN

CSI-2 v2.1SCR EN

DPCM EN

CSI-2 v4.0AOSC

A-PHY SerDes

CSI-2 v4.0AOSC

CSI-2 v4.0 (INT) USL ENLRTE ENSCR_EN

CSI-2 v2.1SCR EN

DPCM EN

CSI-2 v3.0SROI ENUSL EN

CSI-2 v1.3SCR ENRAW 12

CSI-2 v4.0

CSI-2 v4.0 supports multiple features over 4 modes of transport

STD USL AOSC ADAPL

RAW ملسو هيلع هللا ىلص ملسو هيلع هللا ىلص ملسو هيلع هللا ىلص ملسو هيلع هللا ىلص

DPCM ملسو هيلع هللا ىلص ملسو هيلع هللا ىلص ملسو هيلع هللا ىلص ملسو هيلع هللا ىلص

SCR ملسو هيلع هللا ىلص ملسو هيلع هللا ىلص ملسو هيلع هللا ىلص ملسو هيلع هللا ىلص

LRTE ملسو هيلع هللا ىلص ملسو هيلع هللا ىلص X ملسو هيلع هللا ىلص

SROI ملسو هيلع هللا ىلص ملسو هيلع هللا ىلص ملسو هيلع هللا ىلص ملسو هيلع هللا ىلص

FSAF ملسو هيلع هللا ىلص ملسو هيلع هللا ىلص ملسو هيلع هللا ىلص ملسو هيلع هللا ىلص

ISEC ملسو هيلع هللا ىلص ملسو هيلع هللا ىلص ملسو هيلع هللا ىلص ملسو هيلع هللا ىلص

STD – Unidirectional C/D-PHY, CCI over I2C / I3C / SPI, GPIOsUSL – Encapsulation using bidirectional C/D PHYAOSC – Encapsulation using bidirectional I3CAPHY – Integrated and Discrete SerDes transport


• New automotive white paper to be published this week• A-PHY to support broadest spectrum of speed needs using two profiles • Auto-related features included in CSI-2 v3.0, as well as next camera and display

specifications • Now identifying requirements for A-PHY v2.0

Recent activities:


• Information on new features and specification releases

• Use cases and applications of MIPI specifications

• Q&As with working group chairs and other experts

• Latest MIPI Alliance news

• Highlights and key takeaways from webinars and MIPI DevCon presentations

• Details of MIPI participation in industry events

In the blog:

mipi.org/blog

http://resources.mipi.org/blog


Summary• Resolve problems that cannot be solved by single company

• Advance provisions to enable machine awareness through sighto Methodical phased development: Mobile | Client | IoT | Automotive

• Democratize benefits of AI vision through standardization

• Together we go further o PHY WG: A/C/D-PHYso SWG: I3Co SWWG: DisCo for Imaging o Liaisons: DMTF | ETH/WiFi


Backup Material


Virtual Channel Expansion

URL

System Considerations

• Proliferation of Image Sensors Require CSI-2 Aggregators and VC Expansion

• System is limited to 16 or lower VCs:– Supported by CSI-2 over C/D-PHY

• System requires more than 16 VCs:– Supported by CSI-2 over C-PHY

URL

http://assets.inhabitat.com/wp-content/blogs.dir/1/files/2016/09/Uber-Self-Driving-Car-lead-1580x530.jpg

http://libertyviral.com/wp-content/uploads/2016/08/uber-self-driving-diving.jpg


Mobile

Target Platform: Smart Phones, Tablets

Target Imaging Application: Pristine still photography l

streaming video

Client & IoT

Target Platforms: Content Creation (AIO, Notebooks, Hub)

Target Imaging Application: Comprehensive E2E imaging conduit to enable machine

awareness

Autonomous

Target Platform: Automotive, Drones, Medical, Factory

Target Imaging Application: Real time perception and

decision making

IP Reuse / Engineering

Imaging System Platforms


Evolving CSI-2 Architecture for Automotive

CSI-2 Vision

Long Range Radar

Video Camera / LiDAR

UltraSonic

CSI-2 Aggregators may benefit from Virtual Channel Expansion and LRTE PDQ

CSI-2 streaming data from aggregated imagesensors for near real-time perceptionand decision making platform solutions


Deep Neural NetworksMyriad X Hybrid Arch Deep Neural Network 8 Cameras using 16 MIPI Lanes - 700 MP/s

Drone Collision Avoidance (DJI SPARK)

http://www.electronicdesign.com/industrial-automation/intel-s-myriad-x-vision-chip-incorporates-neural-network

evolution of mipi imaging conduit...with certificate signing • data signing -data is originating...

Documents