Slide 1

ExoGENI RacksIlia Baldine ibaldin@renci.orgTestbed14 GPO-funded racksPartnership between RENCI, Duke and IBMIBM x3650 M3/M4 servers1x146GB 10K SAS hard drive +1x500GB secondary drive 48G RAM 1333MhzDual-socket 6-core Intel X5650 2.66Ghz CPUDual 1Gbps adapter10G dual-port Chelseo adapterBNT 8264 10G/40G OpenFlow switchDS3512 6TB sliverable storageiSCSI interface for head node image storage as well as experimenter sliveringEach rack is a small networked cloudOpenStack-based (some older racks run Eucalyptus)EC2 nomenclature for node sizes (m1.small, m1.large etc)Interconnected by combination of dynamic and static L2 circuits through regionals and national backboneshttp://www.exogeni.net

2

2

ExoGENI Status2 new racks deployedRENCI and GPO2 existing racks (not yet OpenFlow enabled) Duke and UNC2 more racks available by GEC14FIU and UHConnected via BEN (http://ben.renci.org), LEARN and NLR FrameNetPartner racksNICTA (under construction)U of Alaska Fairbanks

3Rack diagram and connectivityRack has a management connection to campus networkIt may have an optional connection to the OpenFlow campus network for experimentsA connection to FrameNet or I2 IONDirectVia a pool of vlans with static tags4Rack IP address assignment/24 of publicly routable IP addresses is the best choice2 are assigned to elements of the rackManagement/Head nodeSSG5 VPN appliance (to create a secure mesh for management access between racks)The rest is used to assign IP addresses to experimenter instancesVMs and hardware nodes5Example rack connection (GPO/BBN)6Rack softwareCentOS 6.X base installResource ProvisioningxCAT for bare metal provisioningOpenStack + NEuca for VMsFlowVisor NOX used internally by ORCA GENI SoftwareORCA for VM, baremetal and OpenFlowFOAM for OpenFlow experimentsWorker and head nodes can be reinstalled remotely via IPMI + KickStartWorking on security related to updatesMonitoring via Nagios (Check_MK)ExoGENI ops staff can monitor all racksSite owners can monitor their own rackSyslogs collected centrally

7Rack Software Stack8Rack installationParticulars:Power options include (negotiated ahead of time)208V 3Phase208V 1Phase120V 1PhaseTotal of ~10kW of power needed.Space:e1350 42U Rack Cabinet 79.5" H x 25.5" W x 43.5" D (2020 mm x 648 mm x 1105 mm)Racks arrive on-site pre-assembled and pre-tested by IBM with most software already pre-installedIBM representative will need to come on-site to complete install and hookupNBD hardware supportExoGENI Ops finishes ORCA configurationGPO acceptance testing

9ExperimentationCompute nodesUp to 100 VMs in each full rackA few (2) bare-metal nodesTrue Layer 2 slice topologies can be created Within individual racks Between racksWith automatic and user-specified resource binding and slice topology embeddingOpenFlow experimentation Within racksBetween racksInclude OpenFlow overlays in NLR (and I2)On-ramp to campus OpenFlow network (if available)Experimenters are allowed and encouraged to use their own virtual appliance images

10

10ExoGENI slice isolationStrong isolation is the goalCompute instances are KVM based and get a dedicated number of coresCaveat: currently all instances get 1 core (different RAM and disk). Will be remedied by Summer/Fall 2012VLANs are the basis of connectivityVLANs can be best effort or bandwidth-provisioned (within and between racks)Caveat: current hardware in the racks allows best-effort VLANs only will be remedied by Fall 2012 with support from the vendor11ORCA OverviewOriginally developed by Jeff Chase and his students at DukeFunded as Control Framework Candidate for GENIJointly developed by RENCI and Duke for GENI since 2008.Supported under several current NSF and DOE grants to enable ORCA to run computational networked cloudsFully distributed architectureFederated with GENIWe do not run SAs or issue GENI credentialsWe honor GPO and Emulab-issued credentialsSupports ORCA-native interface, resource specification and toolsFlukesSupports GENI AM API and GENI RspecOmni(Almost) compatible with Gush12ORCA Deployment in ExoGENIEach rack runs its own SM actor that exposesORCA native APIGENI AM APIRack-local SMCan only create slice topologies with resources within that rackExoSM has global visibilityHas access to resources in all racksHas access to network backbone resources for stitching topologies between racks13ORCA deployment14The teamGrand Pooh-bah Jeff ChaseExoGENI OpsBrad Viviano (RENCI) rack hardware designChris Heermann (RENCI) rack networking designVictor Orlikowski (Duke) software packaging and configurationJonathan Mills (RENCI) operations and monitoringORCA DevelopmentYufeng Xin (RENCI) Aydan Yumerefendi (Duke)Anirban Mandal (RENCI)Prateek Jaipuria (Duke)Victor Orlikowski (Duke)Paul Ruth (RENCI)15

1 or 10G

bps

1Gbp

s re

mot

e m

anag

emen

t an

d iS

CSI s

tora

ge c

onne

ctio

n

To campus Layer 3 Network

Dataplane to dynamic circuit backbone (10/40/100 Gbps)

Worker node (IBM x3650 M3)

Worker node (IBM x3650 M3)

Worker node (IBM x3650 M3)

Worker node (IBM x3650 M3)

Worker node (IBM x3650 M3)

Worker node (IBM x3650 M3)

Worker node (IBM x3650 M3)

Worker node (IBM x3650 M3)

Worker node (IBM x3650 M3)

Worker node (IBM x3650 M3)

Management node (IBM x3650 M3)

Dataplane to campus OpenFlowNetwork

Management switch (IBM G8052R)

Sliverable StorageIBM DS3512

OpenFlow-enabled L2 switch (IBM G8264R)

VPN Appliance (Juniper SSG5)10/1001 Gbps Campus

1 Gbps

Juniper SSG5

IBM G8264R

NOX

1/10GbE

VPN appliance

OpenFlow switch

10/100

NLRFrameNet

PoblanoNEC IP8800

Hybrid mode

IBM x3650 M3 - 2.40GHz, 12GBHead node

1/10GbE

Internet

KomarCentOS 5 1GbE

BBN

IBM G8052RManagement switch

1GbE

2601-2610

Bare-metal provisioning (XCat)

EC2 Virtual Cloud(OpenStack)

ORCA

Monitoring(Nagios)

GENI I&M ExoGENI Layer

System Layer

FlowVisor

OF Switch

ALT-G API GENI I&M API

Experimenter Tools

Experimenter Tools

Testbed Tools

Control planes

BEN, NLR, I2, ESnet, ANI

GENI API

ORCA Handlers

Testbed Substrate

Tool layer

FOAM

ORCA Resource Handlers

StarLight

DOE ANI

ExoBroker

ExoSM

I2 AM ANI AM

BEN

LEARN AM

BEN AM

SL AM

LEARN

OpenFlow

OpenFlow

RCI XO

Future XO

A Broker

A AMA SM

B Broker

B AMB SMI2

OpenFlowC AMC SM

NLR

C Broker

BBN XO

OpenFlow

Future XO

B Broker

B AMB SM2601-2610

Global ExoGENI Actors (geni.renci.org, geni-net.renci.org, geni-ben.renci.org)

rci-hn.exogeni.net

bbn-hn.exogeni.net

NLR AM

Campus OF

Campus OF