data center interconnects - lightcounting center interconnects.pdf · lightcounting market research...

LIGHTCOUNTINGMarket Research on High-Speed InterconnectsDatacom, Telecom, CATV, FTTX, Consumer markets © Copyright 2012 LightCounting LLC All material proprietary and confidential.

Data Center Interconnects

Brad SmithVP & Chief Analyst

Data Center Interconnects

[email protected](408) 813-4345

3/12/2012 1


Virtually all the Internet traffic – goes through an opticaltransceiver somewhere in the telecom and datacom network!

3/12/2012 2

Virtually all the Internet traffic – goes through an opticaltransceiver somewhere in the telecom and datacom network!

Supplied by transceiver companies…About one quarter away for bankruptcy


3/12/2012 3

Virtually all the Internet traffic – goes through anoptical transceiver somewhere in the telecom and

datacom network!

Supplied by transceiver companies…About one quarter away from

bankruptcy!


Waiting for New ServersWaiting for 10GBASE-T

Preparing for the Exa Flood of Data

3/12/2012 4

State of the Data Center Business


State of the Industry• Data centers are largely very cost sensitive; governs all

decision making– Decision algorithm (Cost of new product) / Server cost– Buy 100G LR4 transceiver OR 8 servers?

• Average and large data center needs 4-5G, not 10G yet• Getting ready for 10G and 40G QSFP• Most popular optical transceiver = 1G SFP+ & 10G SFP+

• Top-of-rack & End-of-row switches are 10G SFP+ & 40G QSFPready but ….

• Entire industry is “Waiting for new Romley servers”• And 10GBASE-T low-power 28-nm parts

3/12/2012 5


Intel’s Tick-Tock ModelSets the Clock for Data Center Upgrades

Tick= Architecture upgrade ; Tock= Silicon Shrink

3/12/2012 6

Delayed by ~8 months (Leap year effect?)Data center spending delayed by 8 months


MPUs – Key Data center Driver

• 8cores, 16 threads (AMD 16 Cores)• Increase DRAM support 128G to 768G+• PCI Express 3.0 faster I/O• Energy efficient• Bottom line: 80 percent more performance than the

previous generation Xeon 5600 MPUs

3/12/2012 7

X540 Lan-on-motherboard (LOM) comes in at only 25% ofthe cost of an SFP+ fiber per port deployment.


Microprocessors

3/12/2012 8

2-4 Cores 10-16 Cores 40-50 Cores

2 MPUs/Server128GBytes DRAMPCIe Gen2 (8 lanes @5G/lane

? MPUs/Server1T+ Bytes DRAMPCIe Gen3 (8G/lane)ARM, INTEL, AMD

4 MPUs/Server768G Bytes DRAMPCIe Gen3 (8/16 lanes @8G/lane

?Data centerin a server


I/O for Interconnects• Virtualization software

– Sends jobs to non busy MPUs– Increases utilization from 20% to 90%

• I/O increases: Data Direct I/0 - transfer of data directly to from thenetwork interface and cache, bypassing main memory

• PCI Express Gen 2 5G 8b/10b and 8 lanes• PCI Express Gen 3 8G 128/132 and 8-16 lanes• “66 GB/s in tested I/O bandwidth”

• Bottom line: ~300 % faster I/O bandwidth!

3/12/2012 9

20% 90%


Servers per Rack

3/12/2012 10

48 - Severs / Rack60+ Severs / Rack


Cabling in the “Marketing” Brochures

3/12/2012 11


3/12/2012 12

Traditional Data Center Reality


3/12/2012 13

Cabling Realities


The Yellow Wall

3/12/2012 14


Traditional Data centerComputer & Switch History Museum

3/12/2012 15


Modern Data center

3/12/2012 16


3/12/2012 17

.primecaretech.com

Mega Data Center


SuperComputer Site (HPC)

3/12/2012 18

16M Watts of power


Apple South Carolina Data Center

3/12/2012 19


Most High-Speed InterconnectsApproaching the “10G” Speed Barrier

Most High-Speed InterconnectsApproaching the “10G” Speed Barrier

One of the biggest technical transitions in the communicationsindustry and huge opportunities for photonic interconnects


Major Data Center ProtocolsAll Hit 10Gbps barrier

• Direct Attach: 10G = 7m; 14G=3m; 25G will need active DAC?• 10GBASE-T needs microprocessor sized PHY to hit 10G

• InfiniBand: 10G moving to 14G FDR & 28G EDR• Ethernet: 10G moving to 25G• Fibre Channel: 8G moving to 14G & 28G• SAS: 6G moving to 12G and 24G• PCI Express: Gen2 5G moving to Gen3 8G & Gen4 16G

• All pass ~10Gbps where copper has issues• Reaches are becoming longer

• Optical value proposition = high data rate + long reach• Many data center protocols transitioning fro Copper-to-Optical

3/12/2012 21


Smartphones & Tablets• Battle is on features:

– HDTV “Blue-ray” 1080p video– 8Mpixel cameras 24 M pixel cameras

with digital zooms– 4 core microprocessors; 64-128GB FLASH Ram– 2Kx1K resolutions “retina” displays & screen size

Next wave– iMoney – smartphone credit cards– Video calls on 4G networks and WiFi

3/12/2012 22


Mobile Master Clock

3/12/2012 23

March September


Master Industry ClockIntel & Apple

3/12/2012 24

SmartphonesiPhone Google Android, etc

Intel Tick-Tock ModelProtocol roadmaps- Ethernet- Fibre Channel- SAS- PCI Express

Client SideServer Side

WirelessCATV

TelecomInfrastructure

FTTA

CATVDSL

Long HaulMetroLoops


Consumer Electronics Show


Getting Ready for 25G (100G)Devices, MSAs and Protocols

Silicon photonics

SystemsCabling

TransceiversSemiconductors

3/12/2012 26


MSA Madness!• CFP• CFP/2• CFP/4• CFP/4 DD

• zQSFP, zCXP• eQSFP, eCXP• QSFP2, CXP ext.

3/12/2012 27

It is all digital ones and zeros!OEMs have to supportboth new & old!


Explosion of Protocol for OpticalCopper-to-Optical Transition

3/12/2012 28

Road Maps

InfiniBand

PCI Express

HDMIDisplayPortUSB


M&As Restructure the Industry~24-months

SystemsCabling

TransceiversSemiconductors

3/12/2012 29


Cabling Companies Buy AOC Startups + BUs

• TE bought Zarlink BU• FCI Bought MergeOptics• Molex bought Luxtera BU• Samtec bought AlpenIO

Result:• $ xx Billion, international connector, cabling, companies• Offering transceiver and AOCs, EOM, VCSELs

– New set of competitors for transceiver companies– Full line of copper & optical products– Power cables, every protocol, every MSA, connector, etc

3/12/2012 30


Cisco Introduced Blade ServersCompetes against former server partners

• HP bought 3Com/3Par• IBM bought Blade Network Technologies• Dell bought Force10• Oracle bought Sun Microsystems• Mellanox bought Voltaire (InfiniBand)• Intel bought QLogic and Fulcrum (InfiniBand)

Result:• Integrated server & switch companies• Expect new architectures• Expect integrated server & switch products

3/12/2012 31

“Rack as a Sku”


Cisco Buys Transceiver CompaniesCoreOptics & LightWire

• CoreOptics: 100G Long haul in-house• LightWire: Silicon photonics in-house

– 4x25G (100G) -- WDM– High speed modulation (40G) & integrated electronics– Data center (500 meters) & Metro (10Km)

Result:• Servers, switches, and interconnects – all in-house

3/12/2012 32


What’s Next?• Start a wave of M&As of silicon photonics and transceiver

companies?– Yes (p=70%) Who? Companies with innovative 100G-400G technologies.

• M&As wave of core component companies?• 25G VCSELs, TIA/LD, semiconductor companies?

LightCounting speculation:1) Expect new architectures blends servers and switches, ARM?2) “Rack as a Sku”3) Key-locked port concept to increase4) Inside the rack - EOMs and fiber fabrics, embedded AOCs5) Multi-Rack links will be from the OEM (transceivers, AOCs, DAC)6) Open market, unlocked ports; IEEE standard interoperable – for a price!

Bottom line is the companies want to increase their bottom lines

3/12/2012 33

?


The only thing I remember fromCollege English classes

3/12/2012 34

In the words of one poet,

“The Big fish eat the Large fish….

the Large fish eat the Tiny fish…..

…….. and the Tiny fish eat ---mud!”


“Waiting for 10GBASE-T• Reeaaaaly big!• “When it gets small … ….. it will be big”

• Lan On Motherboard (LOM) is key to market- PHY chip needs 28-nm (now at 40-nm)- Late 2013/2014

- 10GBASE-T likely the last LOM implementation

- VCs spent $850M on 10+ startups

- 40GBase-T and 4x10GBase-T likely will be from majorsuppliers – not startups!

- When it happens:- It will last for 8+ years 2013+8= 2022!- Likely will crush Direct Attach copper

3/12/2012 35

130-nm - This will be the year!90-nm - This will be the year !65-nm - This will be the year !40-nm - This will be the year!28-nm – This will be the year!


Direct Attach copper• Has become very popular <5-7meters• Filling the 10GBASE-T void• Low-cost, low-power, Plug & Play• Cheaper than optical SFP+• Active & Passive DAC

• Numerous problems surfacing– Electrically linking different vendors equipment together directly

– different than linking optically– “Limiting” and “non-limiting” ports – no warning labels

• Net: “DAC in the Rack”• ---until 10GBASE-T LOM shows up

• Then game is 28G SFP+ and QSFP

3/12/2012 36

10G = 7 meters14G = 3meters28G = ????


40G QSFP (4x10G)40G SR & 40G LR4

• “SFP+” Replacement• Hot product for transceivers & AOCs next 2 years• All the Top-of-rack and End-of-row switches are 40G ready but ….

“Waiting for Intel Romley servers”

Meets User Wants and Needs• 1 slot – 1 MSA – Multiple reaches – MMF & SMF

– SR4 (100 meter) or LR4 (10Km)– Double SMF or 4 SMF

• Influx of new vendor announcements (USA, J, Asia, Euro)• Prices are going to drop rapidly $300 <$200

• Net: 40G QSFP is where the money will be from 2012-2014 withRomley/Ivy Bridge-class servers

3/12/2012 37


40G QSFP• 2 Phases to market adoption

– Phase 1: Linking Top-of-rack and End-of-rowswitches

• East-West links

– Phase 2: Server uplinks (Intel Ivy bridge MPUs PcieGen3) for high-bandwidth needs

• North-South server links

By the time 10GBASE-T gets its act together, 10G willseem like 56Kb modem

3/12/2012 38


100G

• Lots of discussion and excitement• Google & 10x10MSA woke everyone up to the

coming exa-flood of data•• What customers want: inexpensive, QSFP

solutions (or CFP/4), dual and parallel fibers

• Waiting for 25G VCSELS for short reach• Waiting for LR4 prices to come down

3/12/2012 39GET Luxtera 100G IEE SLIDE


Why 100G Needed in Data Centers100G Driving Forces

• 48 – 60 Servers/ rack• 10Gbps server uplinks

• 480G -600G problem!

• Will demand 100G uplinks

• Timing: Romley Q2/2012 : Volume 2H/2012

• ~ 2H/ 2013 need for 100G transceivers climb rapidly

3/12/2012 40


3/12/2012 41

.primecaretech.com

Mega Data Center

Link this rack

To this rack

400-600metes


100G

• Early shipments just begun shipping• Drivers: Hazwell & Ivy Bridge class Servers & PCI

Express Gen 3 2015+– SR: 40G & 100G shipments 8,000-10,000 (QSFP, CXP)– LR4: 40G 1,800 units & 100G ~ 2,600 units– 10x10MSA ~ 3,500-5,000 units?

• 2Km is really 600 meters – HUH?

3/12/2012 42


Active Optical Cables

• Very popular in High Performance computers (HPCs) TOP25• Technology is proven out and is rippling down to Top 25-100 & High-

end data centers• Used to connect top-of-rack switches to end-of-row switches (4x10G

QSFP)• Large router companies are seriously investigating using 12x10G CXP• Advantages: Plug & Play; green light; No optical connector cleaning,

familiar copper connector; low power (700mW per end)

• InfiniBand today but slowly moving to Ethernet, SAS, PCI Express

3/12/2012 43


Chinese AOCs• Flood of Chinese manufacturers chasing Chinese

Supercomputer deals• Severe price drops

3/12/2012 44


3/12/2012 45

CES AOCs


1 Core AOC TechnologyCreates Multiple Products

• Multiple types– 1x10G SFP+ - 4x5G CX-4– 4x10G QSFP - 4x14G QSFP– 12x10G CXP - 12x12.5G CXP– 8 x 8G iPass

• Breakout AOCs: 4-SFP/QSFP and 3-QSFP/CXP

• Embedded Optical Module (EOM) replacing SNAP12– For inside chassis applications (no MSAs yet)

• Parallel Optical Transceivers

• AOC/Transceiver: AOC with connectorized fibers

3/12/2012 46

Avago

FCI

GigaLight

AlpenIOPhotonera


Embedded Optical Modules“Avago MicroPOD-Like Thingies”

• Inside chassis applications• Mid-board optics• Start out at 10G but the big market will be at 25G• Start out in large routers; later move to servers

3/12/2012 47

Finisar

Avago


3/12/2012 48

Avago EOMs


Optically Enabled Hybrid Chip Packages

• FTTChip– Mez cards– Chip/Optical

3/12/2012 49

Reflex/Fulcrum


SmartPhones and Tablets Interconnects


Tablet

3/12/2012 51


8-Inch AOCs?Ultra-Short Reach

• iPad4, iPhone 6, 2014 class notebooks will haveserious RFI - EMI issues

• HDTVs, high-end video cameras, and othercommercial and consumer electronics will have thesame problem

3/12/2012 52

– Mobile Interface and MediaInterconnect (MIPI.org)

Hitachi Cable


Data center Optics

• Pluggable transceivers

• Active Optical Cables

• Embedded Optical Modules

• FTTChip– Mez cards– Chip/Optical

3/12/2012 53

Finisar

Reflex/Fulcrum

Avago

More high-speed opticswithin and between racks

will push more traffic to 40G/100Guplinks


3/12/2012 54

www.LightCounting.comhttp://www.lightcounting.com/newsletter.cfm


3/12/2012 55


3/12/2012 56

LightCounting.com/newsletter.cfm

Sign up for our free market andnews analysis newsletterLightCounting.com/news.cfm


3/12/2012 57


3/12/2012 58


3/12/2012 59

Our Website:www.lightcounting.com

Headquarters Location:

858 West Park StreetEugene, OR 97401Tel: 408.962.4851Email: [email protected]

[email protected]

data center interconnects - lightcounting center interconnects.pdf · lightcounting market research...

Documents