mipi devcon 2016: multiple mipi csi-2 cameras leveraging fpgas
TRANSCRIPT
Multiple MIPI CSI-2℠ Cameras
Leveraging FPGAs Ted Marena
Director of SoC FPGA Marketing, Microsemi
Agenda • History & adoption of MIPI CSI-2 image sensors • MIPI CSI-2 interface primer • FPGA usage models • Applications for multiple MIPI CSI-2 image sensors with
FPGAs • Summary
2
Evolution of Image Sensor Interfaces
ParallelCMOSinterface subLVDS/HiSPiinterface/etc.
In90s&2000s,parallelinterfacewasthenormInthenextdecade,variousproprietaryinterfaceswereintroduced
~2-3MP
As mobile platforms exploded, so did CSI-2
4
Because of mobile popularity, everyone drifted to CSI-2
• Just like a decade ago when PC components were used broadly, as mobile adoption exploded, so did the acceptance of MIPI CSI-2
5
Title & Date
“LPMode”–1.2VCMOS“HSMode”–SLVS200;Vcm=200mV&Vdiff=+/-100mV
MIPI CSI-2 D-PHY Overview
• Waveforms
LowPowerMode:
TransiUonfromLowPowertoHighSpeed
HighSpeedMode
TransiUonfromHighSpeedtoLowPower
LP11,LP01,LP00,HS0,HS-SYNC….HS0,LP-11
ReferencesthePandNchannelvalueinLPmoderespecUvely
ReferencesthevalueoftheSLVS200signalstate
…00011101=‘hB8
MIPI CSI-2 Transition
MIPI Byte Packet Signaling
8’hB8
8’hB8
4-laneexample
byte_data[7:0] {VC,DT} Byte(0) Byte(4) CS ECC Byte(n)…
…byte_clk
sync
byte_data[15:8] WC Byte(1) Byte(5) {VC,DT} Byte(0) Byte(n-1)…
8’hB8 WC Byte(2) Byte(6) WC Byte(1) CS…
8’hB8 ECC Byte(3) CS WC Byte(2) CS…byte_data[23:16]
byte_data[31:24]
8’b00011101HS-SYNCSequence
PacketHeader
VC–VirtualChannelnumberDT–DataTypeWC=16bitwordlengthcount
ECC–ChecksumonwordcountCS–Checksumondata
MIPI D-PHY, CSI-2 Overview
• Universal D-PHY
Title & Date
1.2VCMOSTransminer
1.2VCMOSReceiver
1.2VCMOSContenUonDetecUonCircuit
SLVS200Transminer
SLVS200Receiver
DynamicTerminaUon
D-PHY vs. C-PHY
• The majority of MIPI CSI-2 cameras use the D-PHY.
C-PHY,3-wire,embeddedclock,3-phasecoding
D-PHY,diff-pairs,sourcesynchronous,nocoding
TransiUonateverysymbol,usedforclockrecovery
Emulating D-PHY in FPGA • Normally FPGAs do not have a D-PHY hard I/O block • They usually interface via translation resistors
11
FPGAI/OsImageSensorI/Os
Key Blocks Used in FPGAs for Imaging
12
AP/uP/uC
MathBlocks/DSP MemoryBlocks
I/OGearing Processor/Micro
FPGAs in Imaging/Video Applications
Bridging is the simplest design.
Acceleration requires more performance & capabilities.
13
FPGAA B FPGA
HWACCA A’
DRAMOp9onal
BridgingAcceleraUon
FPGAs Also Do Processing & Aggregating
Processing could be in an embedded processor or with FPGA fabric, memory and math blocks (DSP blocks).
Aggregation leverages the large I/O capability of FPGAs and the fabric.
14
FPGAAggrega9on
A
A’
DRAM
Processing AggregaUon
FPGAProcessing
A A’
B’
DRAM
B
C
Why Use an FPGA? • Not enough MIPI CSI-2 or other inputs on an AP/ISP • Limited numbers of ISP engines on an AP/ISP • FPGA may perform some processing allowing for a lower
cost AP/ISP • Newer capabilities that are not available with an AP/ISP • An FPGA can implement a complete ISP
15
Multi MIPI CSI-2 Camera Applications • 3D camera / virtual reality • Dual surveillance • Multiple-image sensor HDR • 180-degree surveillance • 360-degree panorama • Surround view automotive • Depth-detection applications • Drone usage
16
3D Camera Example
DualImageSensors
Bridge
CSI-2 CSI-21-4DataLanes
Dual CSI-2 Bridge
CSI-21-4DataLanes
Syncandformatlogic
AlthoughISPdevicesorenhavemulUplecamerainputs,theyorenbenefitfromanFPGA.Synchronizingtheimagesensors&arrangingthem
3D Camera FPGA Implementation
DualImageSensors
Bridge
CSI-2 CSI-21-4DataLanes
Dual CSI-2 Bridge
CSI-21-4DataLanes
Syncandformatlogic
TheFPGAcanarrangetheimageinaside-by-sideoratop-bonomconfiguraUon.ThismakesiteasierfortheISPorAPtoprocesstheimage.
Dual Surveillance Camera
DualImageSensors
Bridge
CSI-2 CSI-21-4DataLanes
CSI-21-4DataLanes
CSI-2Tx
AlthoughISPdevicesorenhavemulUplecamerainputs,theyorenbenefitfromanFPGAthatcanarrangetheimage.
ISP
ImageBufferMemory
Displayspliter
Dual Surveillance FPGA Function
ThissoluUonallowsanISPtoprocessbothimages.
• BothimagesarerecoveredintheFPGA• TheFPGAcombinesthetwoimagesinto
onefortheISP–orenatop-bonomconfiguraUon
• ThisallowstheISPtoprocessthetwoimagesasone,buttheoutputcanbesplitintotwoimages
Image Sensor HDR Processing
CSI-2 May go to ISP/AP or storage
Each image sensor captures frames at exactly the same time. A short, medium and long exposure is used for each.
ImageBufferMemory
FPGAwithHDRengine
Image Sensor HDR Processing
CSI-2
Short-, medium- and long-exposure images. Processed using local and global tone mapping, motion artifact correction, etc.
ImageBufferMemory
FPGAwithHDRengine
180-Degree Surveillance Camera
Bridge
CSI-2 CSI-21-4DataLanes
CSI-2 Stitching
CSI-21-4DataLanes
CSI-2
TheimagesUtchingfuncUonismoreeasilydoneinanFPGA.Inthisdesign,theenUreISPcouldalsobeintheFPGA.
ISP
ImageBufferMemory
.
.
.3 or more
180-Degree Surveillance FPGA Function
• MulUpleimagesarerecoveredintheFPGA
• Theframesarestoredlikelyinexternalmemory
• TheFPGAperformsananalysistodeterminewheretomergetheimage
• TheimagesaresUtchedtogether• Alsolikelyasmoothingtechniqueisused• Theimageoutputisthenprocessedin
theFPGAorformanedandpassedontotheISPorAP
360-Degree Cameras
CSI-2
Each image sensor frames are captured and combined. Image processing could be in the FPGA or AP/ISP.
ImageBufferMemory
FPGAwithsUtching,fisheye
correcUon,etc
.
.
.
ISPorAP
360-Degree Cameras • The FPGA performs an analysis to determine
where to merge the images • The images are stitched together • Depending on the output format, fisheye
correction may be implemented • The image output is then processed in the FPGA
or formatted and passed onto the ISP or AP
26
Surround View Automotive
CSI-2
FrameBuffermemory
FPGA
FPD3LinktoMIPICSI-2
FPD3LinktoMIPICSI-2
Surround View Application • The FPGA implements the stitching of the images • It formats the image for the ISP/AP • Or FPGA processes the image and drives the display • FPGA could add overlay such as directional lines
Multi Camera for Depth Detection
CSI-2
The FPGA processes all the images and provides the lowest possible latency for highest accuracy & response.
ImageBufferMemory
FPGAsyncsdualcamerapairs&doestheprocessing
.
.
.
CSI-2
Couldbeoneormorepairs
Multi Cameras for Depth-Based Analytics DualImageSensors
TheFPGAorAPcomputes3Dpointcloudfromatop-downstereopair.Depth-basedanalyUcscandisUnguishadults,kids,peoplefromshoppingcartsforaccuratepeoplecounUng.
Top-DownDepthImage
FlowMetrics™byPercepTonic
Depth Detection for Robotics • The FPGA synchronizes each camera pair and processes what each
camera pair sees • The robot can preciously determine the distance for each axis that
has a camera pair • Parallel processing of the FPGA gives the robot the quickest
response & accuracy
Multi Camera Drone Application
CSI-2
These dual cameras allow a drone to “see.”
ImageBufferMemory
FPGAsyncsdualcamerapairs,muxes,encrypUon,
etc....
CSI-2
GigE/10GE
Encryptedvideo(wifiorotherRF)
PCIe
Multi Camera Drone Application • The FPGA can merge, mux, pre-process, and run
analytics pre-processing before the AP gets involved for 3D point cloud.
Adds situational awareness so that drones do not crash into anything.
Summary • Multiple camera applications continue to grow. • MIPI CSI-2 adoption will grow and be further adopted
outside of mobile applications. • An increasing number of FPGAs will support MIPI CSI-2
image sensors. • The parallel architecture of the FPGA is ideal for multiple
camera applications.
34