application centric infrastructure (aci), the policy driven data centre

© 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public

Application Centric Infrastructure (ACI), the Policy Driven Data Center

Mike Herbert - Principal Engineer, Cisco Dave Cole, Consulting Systems Engineer, Cisco Sean Comrie, Technical Solutions Architect, Cisco


House Keeping Notes

• Thank you for attending Cisco Connect Toronto 2015, here are a few housekeeping notes to ensure we all enjoy the session today.

•  Please ensure your cellphones / Laptops are set on silent to ensure no one is disturbed during the session

•  A power bar is available under each desk in case you need to charge your laptop


•  Cisco dCloud is a self-service platform that can be accessed via a browser, a high-speed Internet connection, and a cisco.com account

•  Customers will have direct access to a subset of dCloud demos and labs

•  Restricted content must be brokered by an authorized user (Cisco or Partner) and then shared with the customers (cisco.com user).

•  Go to dcloud.cisco.com, select the location closest to you, and log in with your cisco.com credentials

•  Review the getting started videos and try Cisco dCloud today: https://dcloud-cms.cisco.com/help

dCloud

Customers now get full dCloud experience!


Evolution of the Data Center


IT Challenges and Opportunities

IT’s ability to deliver innovation

IT’s budget

Nee

d: IT

Sim

plifi

catio

n

Better alignment of IT with rapidly changing business needs requires dynamic and automated policy-based control of DC and Cloud infrastructure.


Switch ASICs

X86 CPUs

2013 2014/15 2015+

28nm 16nm 65nm Cisco

40nm 28nm 65nm Others

14nm 22nm Intel

Capacity and Cost – Impact of Mega Scale DC’s


What’s the DNA of your applications ?

7 FUTURE < 2000 2003 2006 2008 2010 2012 2013 2014 2011

?


The on-going “IT pain” •  High cost, heterogeneous systems

•  Redundant functionality

•  Lack of agility to innovate

•  Slow time to market

•  Rising maintenance costs

•  Rising regulatory and compliance costs, multiplied by: •  Heterogeneous systems •  Geographic expansion / local laws

•  Falling IT Budgets

8


What Happened

?


•  Separation of IT areas / buying-centers / silos preventing IT to move at the speed demanded by the business

•  Focus changed from Consolidation to Automation and now to Consumption

•  Business owners and Apps Developers started to go straight to public cloud to meet agility and demand. Security and Data Sovereignty arise.

•  Operations become further relevant. Shift from “what it does / how it works” to “how to use / how to consume it”.

DevOps


App Development via DevOps is Changing the Behavior

DevOps


DevOps: Where does each “tool” fit ?

CONTINUOUS INTEGRATION

CONFIGURATION MANAGEMENT ORCHESTRATION &

MANAGEMENT (O&M)

Infrastructure as Code


… so, let’s talk about the elephant in the room…

Current networks are not inflexible nor expensive. Operational process around them makes them just like that. ACI simplifies IT and becomes an enabler.

“Elephants can dance”.


Abstraction, the real objective of “SDN” How to Avoid Death by Micromanagement

You can not mask complexity with

complexity

Less Networks, Not More


Control & Audit Connectivity (Security – Firewall, ACL, …)

IP Address, VLAN, VRF

Enable Connectivity (The Network)

Application Requirements

IP Addressing

Application Requirements

Application Specific Connectivity

Dynamic provisioning of connectivity explicitly defined for

the application

Application Requirements Application Requirements Redirect and Load Balance

Connectivity IP Address, VLAN, VRF

ACI directly maps the application connectivity requirements onto the

network and services fabric

Why Networks are Complex Overloaded Network Constructs


Why Network Provisioning is Slow Application Language Barriers

Developers

Application Tiers

Provider / Consumer

Relationships

Infrastructure Teams

VLANs

Subnets

Protocols

Ports

Developer and infrastructure teams must translate between disparate languages.


What is ACI


“Users” “Files”

ACI Fabric

Logical Endpoint Groups by Role

Heterogeneous clients, servers, external clouds; fabric controls

communication

Every device is one hop away, microsecond latency, no power or port availability constraints, ease of scaling

Flexible Insertion

ACI Controller manages all participating devices, change control and audit capabilities

Unified Management and Visibility

Fabric Port Services

Hardware filtering and bridging; default gateway; seamless service insertion,

“service farm” aggregation

Flat Hardware Accelerated Network

Full abstraction, de-coupled from VLANs and Dynamic Routing, low latency, built-in QoS

Application Centric Infrastructure Fabric


“Users” “Files”

ACI Fabric

Define Endpoint Groups

Any endpoints anywhere within the

fabric, virtual or physical

Enforce Ingress Fabric Rules

Hardware rules on each port, security in

depth, embedded QoS

Single Point of Orchestration

Different administrative groups use same interface, high level of object sharing

Application Policy Infrastructure Controller (APIC) Create Contracts Between Endpoint

Groups

Port-level rules: drop, prioritize, push to service chain; reusable templates

Service Graph

Single Pass Services Security administrator defines generic templates in APIC, availed to contract creation

All TCP/UDP: Accept, Redirect UDP/16384-32767: Prioritize

All Other: Drop

Policy Contract “Users → Files”

ACI is a Fabric which provides a new communication abstraction model


ACI How to build it and how it works


ACI – Components A Policy Based IP Network

Payload IP VXLAN VTEP

AVS VTEP

APIC - Policy Controller & Distributed Management Information Tree (DMIT)

Physical and Virtual L4-7 Service Nodes

Physical and Virtual VTEP’s (Policy & Forwarding Edge

Nodes)

Proxy (Directory) Services

Physical and Virtual Endpoints (Servers) & VMM (Hypervisor vSwitch)

VTEP

IP Network & Integrated VXLAN

WAN/DCI Services

VTEP

VTEP

AVS VTEP


ACI - Components Logical network provisioning of stateless hardware

22

Outside (Tenant VRF)

App DB Web

QoS

Filter

QoS

Service

QoS

Filter

ACI Fabric

Application Policy Infrastructure Controller

Integrated GBP VXLAN Overlay

APIC


Policy Instantiation: Each device dynamically instantiates the required

changes based on the policies

Application Policy Model: Defines the application requirements (Application Network Profile)

VM

VM VM

ACI – 21st Century Distributed Systems in Action

23

App Tier Web Tier DB Tier

Storage Storage

Application Client

VM

10.2.4.7

VM

10.9.3.37

VM

10.32.3.7

VM VM

•  All forwarding in the fabric is managed via the Application Network Profile •  IP addresses are fully portable anywhere within the fabric •  Security & Forwarding are fully decoupled from any physical or virtual network attributes •  Devices autonomously update the state of the network based on configured policy requirements

APIC


Application Policy Infrastructure Controller Centralized Automation and Fabric Management

Layer 4..7 System Management

Storage Management

Orchestration Management

Storage SME Server SME Network SME

Security SME App. SME OS SME

Open RESTful API

Policy-Based Provisioning

APIC

•  Unified point of Data Center network automation and management:

•  Data Model based declarative provisioning

•  Application, Topology Monitoring, & Troubleshooting

•  3rd party Integration (L4-L7 Services, Storage, Compute, WAN, …)

•  Image Management (Spine / Leaf) •  Fabric Inventory

•  Single APIC cluster supports one million+ end points, 200,000+ ports, 64,000+ tenants

•  Centralized Access to ‘all’ Fabric information - GUI, CLI and RESTful API’s

•  Extensible to compute and storage management


APIC Communicating to the Network

25

•  Infra VRF – Used for inband APIC to switch node communication, non routable outside the fabric currently (Multi-Fabric, Remote Leaf will both allow extension of the Infra VRF - Future)

•  Inband Management Network – ‘tenant’ VRF created for inband access to switch nodes •  OOB Management Network – APIC and switch node dedicated mgmt ports

OOB Management Network

APIC will have: 1.  2 attached to fabric for data 2.  2 for mgmt (OOB) 3.  1 console ethernet port (can be only used

for direct laptop hookup) 4.  CIMC/IPMI ports

Inband Management VRF

Infrastructure VRF

Switch nodes will have: 1.  Inband access to Infra & Mgmt VRF 2.  Mgmt Port (OOB) 3.  Console port

APIC APIC APIC


APIC first time Setup •  APIC one time setup is via UCS console access

•  Cluster configuration •  Fabric Name •  Number of controllers [1..9] •  Controller ID [1..9] •  TEP Address pool [10.0.0.1/16] •  Infra VLAN ID [4093]

•  Out-of-band management configuration •  Management IP address [192.168.10.1/254] •  Default gateway [192.168.10.254]

•  Admin user configuration •  Enable strong passwords (Y/N) •  Password

After first time setup, APIC UI is accessible via URL https://<APIC-mgmt-IP>

APIC


Login Screen


Fabric Initialization & Maintenance

28

•  ACI Fabric supports discovery, boot, inventory and systems maintenance processes via the APIC

•  Fabric Discovery and Addressing •  Image Management •  Topology validation through wiring diagram and systems checks

Loopback and VTEP IP Addresses allocated from “Infra VRF” via DHCP from

APIC

APIC Cluster

Topology Discovery via LLDP using ACI specific TLV’s (ACI

OUI)

APIC APIC APIC



29

APIC bootstrap configuration

1)  APIC Cluster Configuration 2)  Fabric Name 3)  TEP Address space (Infra-VRF) 4)  …

Leaf switch discovers attached APIC via LLDP, requests TEP

address and boot file via DHCP

2

1

Spine switch discovers attached Leaf via LLDP, requests TEP

address and boot file via DHCP

3

All nodes in the same APIC cluster should contain same bootstrap information if they are

intended to form a cluster

4

Fabric can be discovered and initialized from multiple sources concurrently

5

6 Fabric will self assemble starting from multiple APIC sources

APIC Cluster

7

APIC Cluster will form when members discovery each other via Appliance

Vector (AV) APIC APIC APIC


Fabric Initialization & Maintenance Node Identity Policy

•  Assigns ID/Name to switches based on serial number

•  Controls which switches can join the fabric

•  Allows zero touch provisioning of switches

POST: https://192.168.10.1/api/node/mo/uni/controller.xml <fabricNodeIdentPol> <fabricNodeIdentP serial=”TNAX234ZA" name="leaf1" nodeId=”101"/> <fabricNodeIdentP serial=” JNAX234ZZ" name="leaf2" nodeId=”102"/> <fabricNodeIdentP serial=“KLAX234ZZ” name="spine1" nodeId=”103"/> </fabricNodeIdentPol>



31

•  ACI Fabric leverages the same Global Catalogue methodology as UCS, the supported HW/SW matrix, image versioning, …

•  APIC and switch node image management controlled via APIC policies •  Policies control which images should be on which groupings of devices, when the images should be

upgraded/downgraded •  Also control the upgrade process, automatic, manual step by step, …

“All-APICs” APIC Cluster

“All-Leafs”

“All-Spines”

APIC APIC APIC


Policy Upgrade of Fabric •  Catalogue Based Software Management


Policy Upgrade of Fabric •  Automated Software Management of all components


APIC - Unified Management and Visibility

•  APIC creates a single point of orchestration for entire network •  Controls underlying fabric topology, service consumer instances, and their policies •  Application, Network, and Security administrators use a single entity to configure their

devices •  High degree of element reuse and templating between different roles and workflows

•  Embedded Role Based Access Control (RBAC) and change management

•  Audit and event correlation capabilities •  Trace specific network events to prior changes, no more management fragmentation/

unknowns

•  Flexible programmability for any managed device or management system •  XML/JSON for Northbound API •  Python scripting for custom device management


ACI Routed Access with Host Based Granularity


ACI Fabric – Integrated Overlay Decoupled Identity, Location & Policy

•  ACI Fabric decouples the tenant end-point address, it’s “identifier”, from the location of that end-point which is defined by it’s “locator” or VTEP address

•  Forwarding within the Fabric is between VTEPs (ACI VXLAN tunnel endpoints) and leverages an extender VXLAN header format referred to as the ACI VXLAN policy header

•  The mapping of the internal tenant MAC or IP address to location is performed by the VTEP using a distributed mapping database


APIC

VTEP VTEP VTEP VTEP VTEP VTEP


ACI leverages VXLAN IETF Draft for Group Based Policy


Location Independent Forwarding Layer 2 and Layer 3

•  Forward based on destination IP Address for intra and inter subnet (Default Mode) •  Bridge semantics are preserved for intra subnet traffic (no TTL decrement, no MAC

header rewrite, etc.) •  Non-IP packets will be forwarded using MAC address. Fabric will learn MAC’s for non-IP

packets, IP address learning for all other packets •  Route if MAC is router-mac, otherwise bridge (standard L2/L3 behaviour)

IP Forwarding:

Forwarded using DIPi address, HW learning of IP

address

10.1.3.11 10.6.3.2 10.1.3.35 10.6.3.17

MAC Forwarding:

Forwarded using DMAC address, HW learning of

MAC address


10.1.1.10 10.1.3.11 10.6.3.2

Distributed Default Gateway •  ACI Fabric supports full layer 2 and layer 3 forwarding semantics, no changes required to applications or end point IP

stacks

•  ACI Fabric provides optimal forwarding for layer 2 and layer 3

•  Fabric provides a pervasive SVI which allows for a distributed default gateway

•  Layer 2 and layer 3 traffic is directly forwarded to destination end point

•  IP ARP/GARP packets are forwarded directly to target end point address contained within ARP/GARP header (elimination of flooding)

10.1.3.35 10.1.1.10 10.1.3.11 10.6.3.2 Directed ARP Forwarding

10.1.3.35

Location Independent Forwarding Layer 2 and Layer 3


10.1.3.11 10.6.3.2

Pervasive SVI •  Default Gateway can reside internal or external to the Fabric

•  Pervasive SVI provides a distributed default gateway (anycast gateway)

•  Subnet default gateway addresses are programmed in all Leaves with end points present for the specific Tenant IP subnet

•  Layer 2 and layer 3 traffic is directly forwarded to destination end point

•  External Gateway is used when Fabric is configured to provide layer 2 transport only for a specific Tenant

10.1.3.35 10.1.1.10 10.1.3.11 10.6.3.2

External Default Gateway 10.1.3.35

Pervasive SVI’s

10.6.3.2 10.6.3.1 10.1.3.1


Host Routing - Inside Inline Hardware Mapping DB - 1,000,000+ hosts

10.1.3.11 fe80::462a:60ff:fef7:8e5e 10.1.3.35 fe80::62c5:47ff:fe0a:5b1a

•  The Forwarding Table on the Leaf Switch is divided between local (directly attached) and global entries

•  The Leaf global table is a cached portion of the full global table

•  If an endpoint is not found in the local cache the packet is forwarded to the ‘default’ forwarding table in the spine switches (1,000,000+ entries in the spine forwarding table)

Local Station Table contains addresses of

‘all’ hosts attached directly to the Leaf

10.1.3.11

10.1.3.35

Port 9

Leaf 3

Proxy A *

Global Station Table contains a local cache of the fabric endpoints

10.1.3.35 Leaf 3 10.1.3.11 Leaf 1

Leaf 4 Leaf 6

fe80::8e5e fe80::5b1a

Proxy Station Table contains addresses of ‘all’ hosts attached

to the fabric

Proxy Proxy Proxy Proxy


Proxy Scaling Scaled based on number of Fabric NFE’s per chassis

Spine Proxy Total Host Entries in the Mapping DB

Network Forwarding

Engines Per Fabric 9336 200K* 2 x NFE

9504 (6 fabrics) 300K 1 9508 (6 fabrics) 600K 2 9516 (6 fabrics) 1M+ 4

NFE

Fabric Module for Nexus 9504

NFE NFE


NFE NFE


NFE NFE

*9336 maintains a single copy of each host entry in the HW proxy DB, 950x maintains redundant copies sharded across Fabric NFE’s


Proxy Database Adjacencies (APIC GUI)


Proxy Database (Oracle)

Spine-1# show coop internal info global Spine-1# show coop internal event-history oracle-adj <IP>

•  You still have full access to all forwarding, adjacency, ..., information via CLI and debug commands when you want them


Endpoint Repository (APIC GUI)


Multicast repository (on APIC GUI)


ACI Endpoint Tracker Application •  Tracks all attachment, detachment,

movement of Endpoints in ACI fabric

•  Stores activity in open source MySQL Database, allowing query capabilities

•  Provides foundation for visualization and query tools

•  Some questions that could be solved: •  What are all the Endpoints on network? •  Where is a specific Endpoint? •  What was connected last Thursday

between 3:30am and 4:00am? •  What is the history of a given Endpoint?


Using Atomic Counters

•  Detect fabric misrouting, debug & isolate application connectivity issues •  Per-application, per-EP, per-EPG real-time, comprehensive traffic counters •  Example:

•  Configure atomic counters on all leafs to count packets EP1->EP2 •  Any counts NOT on Leaf03 or Leaf06 highlight misrouted packets •  Drill-down to Leaf03, Leaf01 and check routing, forwarding entries

•  Configure via policy in appropriate context

10.1.1.10 10.1.3.12 10.6.3.2 10.1.3.35

EP1

Leaf01 Leaf06

EP2


Heatmap

49


Fabric Traceroute

•  Traditional traceroute does not cover multipath technologies; can’t see devices in overlay network •  ACI Traceroute

•  Accurately represents physical & virtual environments •  Complete path visibility

•  Configured via policy in appropriate context •  Fabric •  Infra •  Tenants

10.1.1.10 10.1.3.12 10.6.3.2 10.1.3.35


SPAN

•  How to span traffic between EPGs? •  Could manually config on each leaf node that has a port in target EPG •  Manually reconfig with every move/add/change

•  APIC automatically pushes span configs to every leaf which needs it •  Configure via policy in appropriate context

10.1.1.10 10.1.3.12 10.6.3.2 10.1.3.35 EPG_A


Troubleshooting Wizard

•  https://www.youtube.com/watch?v=Gm9vvHj3LGM


ACI Improved vPC


vPC Behaviour – Standalone & ACI Differences

vSwitch vSwitch

No vPC Peer Link

Required

Standard vPC ACI Based vPC

Orphan Port

‘No’ Orphan Ports (Single

Homed Servers ‘not’ orphans)

Implicit Uplink Tracking

Hardware Based Recovery for server link failures (no STP no vPC

state updates)


FEX Topology Support Roadmap

6.1(2)I2(3) Future Future 6.1(2)I2(3)

Straight Through (Single Homed) vPC (Dual Homed) EvPC

Active/Standby Teaming

Nexus 9300 Standalone

Nexus 9300 ACI Leaf

11.1(x) - 1HCY15 11.0(1d) - Shipping Future Future


Classical vPC •  In classical vPC host addresses are scoped to a

VLAN

•  Traffic is recovered based on updating the VLAN forwarding topology

•  On loss of all of the locally attached members of the vPC MAC address table is updated to forward frames for the vPC across the vPC Peer Link

N5K-1# sh mac-address-table vlan 101 VLAN MAC Address Type Age Port ---------+-----------------+-------+---------+----- 101 001b.0cdd.387f dynamic 0 Po30 101 0023.ac64.dda5 dynamic 30 Po201 Total MAC Addresses: 4

N5K-2# sh mac-address-table vlan 101 VLAN MAC Address Type Age Port ---------+-----------------+-------+---------+----- 101 001b.0cdd.387f dynamic 0 Po20 101 0023.ac64.dda5 dynamic 30 Po201 Total MAC Addresses: 4

MAC_C

MAC_A

N5K-1 N5K-2

1

3

2


vPC in ACI Fabric •  ACI Leaves support virtual port channels (vPC)

interfaces similar to Nexus (802.3ad port channels with links split across two devices)

•  Differences between ACI vPC and standard vPC

•  No Peer Link is required •  Peer communication happens via the

Fabric •  Path recovery also happens via the Fabric

and not peer link •  CFS (Cisco Fabric Services) is replaced by

IFS (ACI Fabric Services) which is based on Zero Message Queue (ZMQ)

•  Forwarding selection (which peer will forward a frame

•  Within the Fabric the vPC interfaces use an anycast VTEP which is active on both vPC peers

ACI Fabric Services (ZMQ)

Host or Switch

VTEP VTEP

vPC Anycast VTEP

vPC Anycast VTEP


vPC in ACI Fabric •  Traffic is both sourced and destined to the anycast vPC VTEP address from remote Leaves

•  A hardware hash in the spine will determine which of the two peers forwards a specific flow downstream to the attached device (flow hashing between the peers via spine

•  In the event of a downlink failure on one of the peers (all local member ports are down)

1.  A bounce entry is created for the end points reachable via the port channel pointing to the peers VTEP

2.  All MAC/IP to Leaf bindings for the specific vPC are removed from the COOP database and the spine proxy

•  On failure of a peer the remaining Leaf converts all vPC ports to non-VPC local ports

Host or Switch

VTEP VTEP

vPC Anycast VTEP

vPC Anycast VTEP

Traffic within the Fabric is sent to the vPC anycast address


ACI Networking and Policy Terms


Backbone

vPC

vPC

vPC

•  Layer 2 and Layer 3 interoperation between ACI Fabric and Existing Data Center builds •  Layer 3 interconnect via standard routing interfaces,

OSPF, Static, iBGP (Supported) MP-BGP, EIGRP, OSPF (1HCY15)

•  Layer 2 interconnect via standard STP or via VXLAN overlays

vSwitch Hyper-‐V AVS

Connecting the ACI Network Layer 2 and Layer 3

Extend Layer 2 VLAN’s where required

Interconnect at Layer 3


Fabric Infrastructure Understanding Networks and Groups

APIC

Outside (Tenant

VRF)

App DB Web

QoS

Filter

QoS

Service

QoS

Filter

Location for Endpoints that are ‘Inside’ the Fabric are found via the Proxy Mapping DB

(Host Level Granularity)

Location for Endpoints that are ‘Outside’ the Fabric are found via redistributed routes sourced from

the externally peered routers (Network Level Granularity)

‘Outside’ EPG associated with external network

policies (OSPF, BGP, … peering)

Forwarding Policy for ‘inside’ EPG’s defined by associated Bridge Domain network policies


Fabric Infrastructure Understanding Networks and Groups

EP EP

EPG EPG

Application Profile

EP EP

Bridge Domain

EP EP

EPG EPG

Application Profile

EP EP EP EP

EPG EPG

Application Profile

EP EP

Bridge Domain

Tenant

Private Network

Private Network


A Tenant is a container for all network, security,

troubleshooting and L4 – 7 service policies.

Pepsi-Tenant Coke-Tenant

Tenant

Tenant resources are isolated from each other, allowing management by different

administrators.



Private Network 1

Private Network 2

Private Network 1

Private Network 2

Private networks (also called VRFs or contexts) are defined

within a tenant to allow isolated and potentially

overlapping IP address space.

Private Networks



Private Network 1

Private Network 2

Private Network 1

Private Network 2

Within a private network, one or more bridge domains must

be defined.

A bridge domain is a L2 forwarding construct within the

fabric, used to constrain broadcast and multicast traffic.

Bridge Domain 1

Bridge Domain 2

Bridge Domain 3

Bridge Domain 4

Bridge Domain 1

Bridge Domain 2

Bridge Domain 3

Bridge Domain 4

Bridge Domain


EPG Definition

EP EP

EPG EPG

Application Profile

EP EP

EPs are devices which attach to the network either virtually or physically, e.g: •  Virtual Machine •  Physical Server (running Bare Metal or Hypervisor) •  External Layer 2 device •  External Layer 3 device •  VLAN •  Subnet •  Firewall •  Load balancer

Virtual Port, Physical Ports, External L2 VLAN, External L3 subnet



Private Network 1

Private Network 2

Private Network 1

Private Network 2

EPGs exist within a single bridge domain only – they do

not span bridge domains.

Bridge Domain 1

Bridge Domain 2

Bridge Domain 3

Bridge Domain 4

Bridge Domain 1

Bridge Domain 2

Bridge Domain 3

Bridge Domain 4

EPG

End Point Groups

EPG

EPG EPG

EPG

EPG

EPG

EPG EPG

EPG


Mapping the Configuration to the Packet

M/LB/SP Flags Flags/DRE VNID == BD/VRF Source Class ID == EPG

•  ACI Fabric leverages an application centric policy model

•  VXLAN Source Group is used as a tag/label to identify the specific end point for each application function (EPG)

•  Policy is enforced between an ingress or source application tier (EPG) and an egress or destination application tier (EPG)

•  Policy can be enforced at source or destination

Coke-Tenant

Private Network 1

Private Network 2

Bridge Domain 1

Bridge Domain 2

Bridge Domain 3

Bridge Domain 4

EPG

EPG

EPG EPG

EPG


ACI Integration and Connecting to existing Networks


Connecting/Extending ACI via Layer 2

Layer 2

Layer 2

Layer 2

Extend L2 domain beyond ACI fabric - 2 options 1.  Manually assign a port to a VLAN which in turn mapped to an EPG. This extend EPG beyond ACI fabric

(EPG == VLAN)

2.  Create a L2 connection to outside network. Extend bridge domain beyond ACI fabric. Allow contract between EPG inside ACI and EPG outside of ACI

Lets Look at the Links


Connecting/Extending ACI via Layer 2 Bridge any VLAN/VXLAN to any VLAN/VXLAN

71

•  Forwarding is ‘not’ limited to nor constrained by the encapsulation type or encapsulation specific ‘overlay’ network

•  VLAN’s are local to the leaf switch

802.1Q VLAN 10

VXLAN VNID = 5789

VXLAN VNID = 11348

NVGRE VSID = 7456

Any to Any

802.1Q VLAN 50

Normalized Encapsulation

Localized Encapsulation

APIC


VXLAN VNID = 5789

VXLAN VNID = 11348

NVGRE VSID = 7456

Any to Any

802.1Q VLAN 50

Normalized Encapsulation

Localized Encapsulation

IP Fabric Using VXLAN Tagging


•  All traffic within the ACI Fabric is encapsulated with an extended VXLAN header •  External VLAN, VXLAN, NVGRE tags are mapped at ingress to an internal VXLAN tag •  Forwarding is not limited to, nor constrained within, the encapsulation type or

encapsulation ‘overlay’ network •  External identifies are localized to the Leaf or Leaf port, allowing re-use and/or translation

if required

Payload

Payload

Payload

Payload

Payload

Eth IP VXLAN Outer

IP

IP NVGRE Outer IP

IP 802.1Q

Eth IP

Eth MAC

Normalization of Ingress Encapsulation

Connecting/Extending ACI via Layer 2 Bridge any VLAN/VXLAN to any VLAN/VXLAN

72


An Example of Interconnecting and Migrating

Logical Design

HSRP Default GW

VLAN / Subnet

P P VM VM VM

P VM

vPC

N7k

N5k

L3 HSRP

P VM

vPC

N7k

N5k

L3 HSRP

N2k

P VM

N7k

FEX

L3 HSRP

P VM

Cat6500

L3 HSRP

Many Different Physical Designs


Extend the EPG Option 1

VLAN 30 Layer 2

100.1.1.3 100.1.1.5

EPG

100.1.1.7 100.1.1.99

•  VLAN’s are localized to the leaf nodes •  The same subnet, bridge domain, EPG can be configured as a ‘different’ VLAN on each leaf

switch •  In 1HCY15 VLAN’s will be port local

100.1.1.3

BD Existing

App

VLAN 20


Extend the EPG Option 1

Layer 2

VLAN 10

100.1.1.3 100.1.1.5 EPG

100.1.1.7 100.1.1.99

•  Single Policy Group (one extended EPG) •  Leverage vPC for interconnect (diagram shows a single port-channel which is an option) •  BPDU should be enabled on the interconnect ports on the ‘vPC’ domain

100.1.1.3

VLAN 30

VLAN 20

BD Existing

App

VLAN 10 VLAN 10 VLAN 10


Assign Port to an EPG •  With VMM integration, port is assigned to EPG by

APIC dynamically.

•  In all other cases, such as connecting to switch, router, bare metal, port need to be assigned to EPG manually or use API

•  Use “Static Binding” under EPG to assign port to EPG

•  The example assigns traffic received on port eth1/32 with vlan tagging 100 to EPG VLAN 100


Assign Port to EPG VLAN Tagging Mode

•  Tagged. Trunk mode •  Untagged. Access mode. Port can only be in one

EPG •  802.1P Tag. Native VLAN. •  No Tagged and Untagged(for different port) config

for same EPG with current software

•  Assign port eth1/1 with VLAN 100 tagged mode and port eth1/2 with VLAN 100 untagged mode to EPG WEB is not supported

•  Use 802.1P Tag. Port eth1/1 vlan 100 tagged, eth1/2 vlan 100 902.1P Tag

•  VLAN to EPG mapping is switch wide significant


C

Extend the Bridge Domain Option 2

Layer 2

100.1.1.3 100.1.1.5 100.1.1.7 100.1.1.99

•  External EPG (policy between the L2 outside EPG and internal EPG) •  Leverage vPC for interconnect (diagram shows a single port-channel which is an option) •  BPDU should be enabled on the interconnect ports on the ‘vPC’ domain •  L2 outside forces the same external VLAN << fewer operational errors

100.1.1.3

BD Existing

App

EPG Inside

EPG Outside

VLAN 30 VLAN 10 VLAN 10 VLAN 10

VLAN 10

VLAN 20


L2 Outside Connection Configuration Example

•  Step 1. Create L2 Outside connection.

•  Associate with BD. •  Specify VLAN ID to connect to

outside L2 network •  External Bridge Domain is a way

to specify the VLAN pool for outside connection.

•  It is NOT a Bridge Domain.


L2 Outside Connection Configuration Example

•  Step 2. Specify leaf node and interface providing L2 outside connection


L2 Outside Connection Configuration Example •  Step 3. Create external EPG

under L2 outside connection •  Step 4. Create contract

between external EPG and internal EPG


Configure ACI Bridge Domain settings •  Temporary Bridge Domain

specific settings while we are using the HSRP gateways in the existing network.

•  Select Forwarding to be “Custom” which allow

•  Enable Flooding of L2 unknown unicast

•  Enble ARP flooding •  Disable Unicast routing

Tenant “Red”

Context “Red”

Bridge Domain “10”

Subnet 10 EPG-10


Migrate Workloads

Existing Design

HSRP Default GW

VLAN 10 / Subnet A

P P VM VM VM

APIC

EPG “10”

P P VM VM VM

APIC point of view, the policy model

VM’s will need to be connected to new Port Group under APIC control (AVS or DVS).


Complete the Migration Change BD settings back to normal for ACI mode

•  Change BD settings back to default. •  No Flooding •  Unicast Routing enabled.


Migrating Default Gateway to the ACI Fabric

Change GW MAC address. By default, All fabric and all BD share same GW MAC

Enable Routing and ARP flooding


ACI Interaction with STP

BPDU

STP Root Switch

Same L2 Outside EPG

(e.g. VLAN 10)

•  No STP running within ACI fabric

•  BPDU frames are flooded between ports configured to be members of the same external L2 Outside (EPG)

•  No Explicit Configuration required •  Hardware forwarding, no interaction

with CPU on leaf or spine switches for standard BPDU frames

•  Protects CPU against any L2 flood that is occurring externally

•  External switches break any potential loop upon receiving the flooded BPDU frame fabric

•  BPDU filter and BPDU guard can be enabled with interface policy

APIC

BPDU

BP

DU

BP

DU


ACI Fabric Loopback Protection

STP Loop Detection

LLDP Loop Detection

•  Multiple Protection Mechanisms against external loops

•  LLDP detects direct loopback cables between any two switches in the same fabric

•  Mis-Cabling Protocol (MCP) is a new link level loopback packet that detects an external L2 forwarding loop

•  MCP frame sent on all VLAN’s on all Ports •  If any switch detects MCP packet arriving on

a port that originated from the same fabric the port is err-disabled

•  External devices can leverage STP/BPDU

•  MAC/IP move detection and learning throttling and err-disable

APIC

BPDU LLDP

MCP Loop Detection

(supported with 11.1 release)

MCP


C

Managing Flooding Within the BD

Layer 2

100.1.1.3 100.1.1.5 100.1.1.7 100.1.1.99

•  In a classical network traffic is flooded with the Bridge Domain (within the VLAN) •  You have more control in an ACI Fabric but need to understand what behaviour you want

100.1.1.3

BD Multi EPG

EPG App 1

EPG Outside

VLAN 30 VLAN 10 VLAN 10 VLAN 10

VLAN 10

EPG App 2

VLAN 20


Managing Flooding Within the Fabric ARP Unicast

ARP Flooding Disabled (Default)

•  Disable ARP Flooding – ARP/GARP is forwarded as a unicast packet within the fabric based on the host forwarding DB

•  On egress the ARP/GARP is forwarded as a flooded frame (supports hosts reachable via downstream L2 switches)

Firewall Configured as the Default Gateway

ARP


Managing Flooding Within the Fabric ARP Flooding

ARP Flooding Enabled •  Enabling ARP Flooding – ARP/GARP is

flooded within the BD •  Commonly used when the default GW is

external to the Fabric

Firewall Configured as the Default Gateway

ARP


Managing Flooding Within the Fabric Unknown Unicast Proxy Lookup

Unknown Unicast Lookup via Proxy

•  Hosts (MAC, v4, v6) that are not known by a specific ingress leaf switch are forwarded to one of the proxies for lookup and inline rewrite of VTEP address

•  If the host is not known by any leaf in the fabric it will be dropped at the proxy (allows honeypot for scanning attacks)

Unknown Unicast

Proxy

HW Proxy Lookup


Managing Flooding Within the Fabric Unknown Unicast Flooding

•  Hosts (MAC, v4, v6) that are not known by a specific ingress leaf switch are flooded to all ports within the bridge domain

•  Silent hosts can be installed as static entries in the proxy (flooding not required for silent hosts)

Unknown Unicast Flooded

Unknown Unicast

Unknown Unicast Flooded


Managing Flooding Within the Fabric Unknown Multicast – Mode 1 (Flood)

•  Unknown Multicast traffic is flooded locally to all ports in the BD on the same leaf the source server is attached to

•  Unknown Multicast traffic is flooded to all ports in the BD on leaf nodes with a ‘multicast router port’

Unknown Multicast Flooded

Unknown Multicast


Managing Flooding Within the Fabric Unknown Multicast – Mode 2 (OMF ‘or’ Optimized Flood)

•  Unknown Multicast traffic is only flooded to ‘multicast router ports’ in this mode

Unknown Multicast Optimized Flooding

Unknown Multicast


Managing Flooding Within the Fabric Scoping Broadcasts to a micro segment

100.1.1.3 100.1.1.5 100.1.1.7 100.1.1.99 100.1.1.3

EPG B

EPG A

EPG C

100.1.1.72

Traffic Type 11.0(x) Behaviour 11.1(x) Behaviour

ARP Flood or Unicast Flood or Unicast

Unknown Unicast Flood or Leverage Proxy Lookup Flood or Leverage Proxy Lookup

Unknown IP Multicast Flood or OMF Flood or OMF

L2 MCAST, BCAST, Link Local Flood Flood within the BD, Flood within the EPG, Disable Flooding within the BD/EPG


Managing Flooding Within the Fabric Multi Destination Flooding (Supported with 11.1(x) – Q2CY15)

•  Link Level Traffic is either •  Contained within the EPG

•  Contained within the Bridge Domain •  Dropped

•  Security Segmentation for Link Level Traffic

Link Level BCAST

Manage Flooding within

the BD

100.1.1.3

100.1.1.5

100.1.1.7 100.1.1.99

100.1.1.72

100.1.1.52

EPG ‘A’

100.1.1.4

EPG ‘A’ EPG ‘B’ EPG ‘B’

EPG ‘B’


Managing Flooding Within the Fabric Flooding scoped to the EPG

100.1.1.3 100.1.1.5 100.1.1.7 100.1.1.99 100.1.1.3

EPG B

EPG A

EPG C

100.1.1.72

•  Link Local, BCAST & L2 Multicast traffic can be managed on a micro-segment basis •  As an example:

•  EPG A, EPG B & EPG C - Link Level traffic is flooded ‘only’ to the endpoints within the EPG


Extension and Connecting It’s a Network with any VLAN Anywhere

Anycast Default Gateway

10.10.10.8 10.20.20.32 10.10.10.9 10.20.20.33

10.20.20.31 10.10.10.6

Any IP - Anywhere


Application Client

Subnet 10.20.20.0/24

Subnet 10.10.10.0/24

Subnet 10.30.30.0/24

Subnet 10.40.40.0/24

Subnet 10.50.50.0/24

External Networks (Outside)

Redirect to Pre-configured FW


Critical Users (Outside)

Middle Ware Servers

Web Servers

Oracle DB Contract

Redirect to dynamically configured FW

NFS Contract Redirect to dynamically

configured FW

Default Users (Outside)

NFS Servers

Subnet 10.20.20.0/24

Subnet 10.10.10.0/24

Subnet 10.30.30.0/24

Permit TCP any any


Policy can be added gradually starting with what you have today


Simple Policy During Migration - Any-to-Any Configuration

Contracts Provided

Filter Contracts Provided

Contracts consumed

Filter

EPG “VLAN 10” VLAN10 Default ALL ALL Default

EPG “VLAN 20” VLAN20 Default ALL ALL


ALL VLAN 10

VLAN 20

VLAN 30


I want to have a very open configuration with VLAN10 talking to anything (Step 1)

•  Create “Contract” ALL if it doesn’t exist yet

•  Use filter “common/default”


I want to have a very open configuration with VLAN10 talking to anything (Step 2)

•  EPG VLAN 10 provides and consumes “ALL”


Extension and Connecting Dynamic Distributed ACL’s

Permit ACL is applied on all ports between VLAN

10, 20 & 30

10.10.10.8 10.20.20.32 10.10.10.9 10.20.20.33

10.20.20.31 10.10.10.6

All Subnets are allowed to communicate with this policy applied


Later if I want to put an ACL between VLAN 10 and 20

ALL VLAN 10

VLAN 20

VLAN 30

Contracts Provided Filter Contracts Provided Contracts consumed

Filter

EPG “VLAN 10” VLAN10 Default VLAN20 Port 80

EPG “VLAN 20” VLAN20 Default ALL ALL Default



Extension and Connecting Dynamic ACL’s

Dynamic ACL is applied between all endpoints only allowing port 80

10.10.10.8 10.20.20.32 10.10.10.9 10.20.20.33

10.20.20.31 10.10.10.6

Traffic is controlled between VLAN 10 & 20 to HTTP (port 80)


ACI Routing


Backbone

vPC

vPC

vPC vSwitch Hyper-‐V AVS

Connecting via Layer 3

Interconnect at Layer 3

•  Layer 3 interconnect via standard routing interfaces,

OSPF NSSA, Static, iBGP - 11.0(x) FCS OSPF, eBGP, EIGRP & Transit Routing – 11.1(x) (1HCY15)

Border Leaf •  Any leaf can be border leaf •  No limit for number of border leaf in the

fabric

•  L3 interface & sub-interface •  VRF-lite for multi-tenancy •  SVI Interface for L2 and L3 outside connection

on same port


Connecting ACI via Layer 3 - Routing J Steps to Enabling Routing

1.  Active Internal Fabric Route Redistribution (MP-BGP)

2.  Configure Routing Peer and Protocol to external WAN/Core routers

3.  Define which internal networks should be advertised to the outside and via which routing peers

4.  Define the outside policy groups (which external networks should be able to communicate to which internal hosts

Border Leaf Router Peering

109 © 2013-2014 Cisco and/or its affiliates. All rights reserved.

§  Fabric runs MP-BGP between spines and leaves

§  Each L3 out is a separate L3 domain

§  Routes learned from L3 outs are redistributed into BGP on border leaves

§  OSPF domains are not joined via the fabric. Leaf switches are ASBRs

ACI fabric is a transit network, supported with 11.1

OSPF Area 0

OSPF Area 0

Different OSPF domains

ACI Fabric as transit MP-BGP

OSPF ASBR OSPF ASBR


OSPF Area 0

Border Leaf

§  Redistribution of routes into MP-BPG (per VRF)

§  Routes are redistributed from MP-BGP to leaf only if VRF is deployed on that leaf.

Redistribution of routes into MP-BGP BGP RR BGP RR

AS-400

EBGP

Border Leaf Border Leaf Border Leaf Border Leaf

AS-200 OSPF Area 10

IBGP

AS-200 MP-BGP Peering Protocol Peering for VRF1 Protocol Peering for VRF2

Routes redistributed into BGP at border leaf Per VRF

Routes redistributed from MP-BGP to border leaf for VRF 2. VRF 1 routes are not redistributed on this leaf


Manage the Fabric MP-BGP Configuration


MP-BGP in ACI Fabric •  MP-BGP is not on by default. Assign BGP ASN and specify spine nodes as BGP RR

to turn on MP-BGP

•  APIC provisions the rest (BGP sessions, RD, import and export target, VPNV4 address family, route-map for route redistribution etc.)

•  MP-BGP doesn’t carry end point tables(MAC and IP)

MP-BGP sessions with two spine nodes


External Routed Networks (L3outside) Configuration

Tenant

External Routed Networks

L3Outside (l3extOut)

Logical Node Profile (l3extLNodeP)

Logical Interface Profile (l3extLIfP)

BGP Peer Connectivity Profile (bgpPeerP)

External Network Instances Profile (l3extInstP)

L3out Name Private Network association External Routed Domain association Protocol selection (i.e OSPF area)

Node selection Router ID configuration Loopback Interface configuration

Interface selection (routed interface, sub-interface, SVI) IP address configuration Association to protocol policy (authentication, network type, etc)

BGP peer configuration BGP settings Remote AS

Import/Export route control subnets Import security subnets Contracts: (provided, consumed, taboo)


Import and Export Route Control Example

100.1.1.0/24 100.2.2.0/24 100.3.3.0/24

Tenant-1:VRF-1 L3 EPG 1: Import route control: 100.1.1.0/24 100.2.2.0/24

BGP Neighbor

BGP Neighbor

Only prefix 1001.1.0/24 added to MP-BGP MP-BGP table. Tenant-1:VRF-1 >i100.1.1.0/24 >i100.2.2.0/24

Tenant-1:VRF-1 L3 EPG 2 Export route control: 100.1.1.0/24

100.1.1.0/24 100.2.2.0/2 100.3.3.0/24

100.1.1.0/24 100.2.2.0/24

100.1.1.0/24 100.2.2.0/24

100.1.1.0/24


§  Route control is configured at the L3out EPG object (L3extInstP)

§  A “route-map” is created for the L3out.

§  An “ip prefix-list” is created for each L3out EPG (L3extInstP)

Export Route Control Configuration Example


§  Policy control enforcement is enabled per Private Network (VRF) §  If policy control is unenforced for the Private Network all data plane

traffic is permitted between L3out EPGs.

§  If policy control is enforced contracts are required between L3out EPGs to allow transit traffic and between Application Profile EPGs for fabric to L3out traffic.

§  Security Policy is enforced for IP prefixes not L4 ports. §  Filters (L4 port filters) are not supported for L3out EPG contracts

§  Security Policy subnets are configured on the L3out EPGs

Security Policy Control Enforcement


Security Policy Subnet Configuration

Zoning rules are created for Security Import Subnets when contracts are configured between L3 outs


ACI Topologies


Interfacing to WAN/DCI Routing (Planned 11.2, Q1CY16) Extending VXLAN to the PE

Direct Connect from Spine to PE

Web/App DB

•  GBP VXLAN hand off from border leaf to WAN/DCI •  Direct Connection between ‘Spine’ and ASR9K and N7K (ASR1K EC is in progress) •  BGP-EVPN L3 route exchange (Layer 2 post 11.2)

MP-BGP – GBP

VXLAN

DCI OTV/VPLS

WAN

DC Site 2

Client PE

PE

PE

PE

•  Direct connect to Spine with GBP VXLAN to PE •  EPG/VRF == Fabric Scale •  Endpoint and LPM == COOP (LISP DB) Scale

Leaf

VTEP

VTEP

VTEP

VTEP

VTEP

VTEP

Spine RR

RR

Border Leaf

EVPN iBGP


Multi-Fabric Scenarios In-Region ‘and’ Out-of-Region

Fabric ‘A’ Fabric ‘B’

Web/App DB

Web/App


Web/App DB

Web/App

•  In-Region (Same Room, Building, Campus, Metro)

< 10 msec RTT

•  Out of Region Data Centers

> 10 msec RTT


Single Fabric Scenarios Multi-Site (Stretched) Fabric

Site/Room ‘A’ Site/Room ‘B’

•  Single Fabric + Multi-Site •  Single Operational Zone (VMM, Storage, FW/LB are all treated as if it is ‘one’ zone)

•  e.g. Single vCenter with Synchronized Storage •  Interconnect between sites

•  Direct Fiber (40G), DWDM (40G or multiple 10G), Pseudo Wire (10G or 40G)

Interconnect Leaf Nodes

HYPERVISOR HYPERVISOR HYPERVISOR

10 msec. Round Trip


Site ‘A’ Site ‘B’

Multi-Fabric – Current Options L2/L3 Classification

Web1

App1

dB1

Web2

App2

dB2

L2_Outside Classify Based on

VLAN

L3_Outside Classify Based on

Network/Mask

Classify traffic arriving from a remote site (fabric) based on the incoming

VLAN or layer 3 prefix (LPM)

HYPERVISOR HYPERVISOR H Y P E R V I S O R HYPERVISOR HYPERVISOR H Y P E R V I S O R


Site ‘A’ Site ‘B’

Multi-Fabrics – Current Options External Synchronization of Fabric Policy

Symmetrical XML Configuration will

maintain consistent operation between

fabrics

Externally triggered Export and Import between Fabrics is another option to

maintain consistency




Multi-Site Traffic

mBGP - EVPN

Multi-Fabric Extended GBP VXLAN (Target Q1CY16)


mBGP is used to advertise host & network level reachability between fabrics

Central Policy Control to coordinate across

multiple fabrics

VTEP IP VNID Tenant

Packet Group Policy

•  Multiple APIC Clusters (N+1 Redundancy for each Fabric)

•  Single Operational Domain via Hierarchical Controller

•  VXLAN is extended between fabrics (EPG information is communicated between fabrics)

•  VXLAN translation permits independent fabrics while maintaining full policy

VTEP IP VNID Tenant

Packet Group Policy

VTEP IP VNID Tenant

Packet Group Policy


Hypervisor Integration


Hypervisor Interaction with ACI Two modes of Operation

•  ACI Fabric as an IP-Ethernet Transport

•  Encapsulations manually allocated •  Separate Policy domains for Physical

and Virtual

VLAN 10 VLAN 10 VXLAN 10000

Non-Integrated Mode

•  ACI Fabric as a Policy Authority •  Encapsulations Normalized and

dynamically provisioned •  Integrated Policy domains across

Physical and Virtual

APP WEB DB

Integrated Mode

DB

126


vCenter DVS SCVMM

§  Relationship is formed between APIC and Virtual Machine Manager (VMM)

§  Multiple VMMs likely on a single ACI Fabric

§  Each VMM and associated Virtual hosts are grouped within APIC

§  Called VMM Domain

§  There is 1:1 relationship between a Virtual Switch and VMM Domain VMM Domain 1

Hypervisor Integration with ACI Control Channel - VMM Domains

vCenter AVS

VMM Domain 2 VMM Domain 3

127


L/B

EPGAPP

EPG DB F/W

EPG WEB

Application Network Profile

VM VM VM

WEB PORT GROUP

APP PORT GROUP

DB PORT GROUP

Hypervisor Integration with ACI

APIC §  ACI Fabric implements policy on Virtual

Networks by mapping Endpoints to EPGs

§  Endpoints in a Virtualized environment are represented as the vNICs

§  VMM applies network configuration by placement of vNICs into:

§  Port Groups (VMWare), §  VM Networks (Hyper-V) §  Networks (OpenStack)

§  EPGs are exposed to the VMM as a 1:1 mapping to Port Groups, VM Networks or OpenStack Networking.

128


VMWare Integration Three Different Options

+

Distributed Virtual Switch (DVS) vCenter + vShield Application Virtual Switch

(AVS)

•  Encapsulations: VLAN •  Installation: Native •  VM discovery: LLDP •  Software/Licenses:

vCenter with EnterprisePlus License

•  Encapsulations: VLAN, VXLAN

•  Installation: Native •  VM discovery: LLDP •  Software/Licenses:

vCenter with EnterprisePlus License, vShield Manager with vShield License

•  Encapsulations: VLAN, VXLAN

•  Installation: VIB through VUM or Console

•  VM discovery: OpFlex •  Software/Licenses:

vCenter with EnterprisePlus License

129


APIC Admin

VI/Server Admin Instantiate VMs, Assign to Port Groups

L/B

EPG APP

EPG DB

F/W

EPG WEB


Create Application Policy

Web Web Web App

HYPERVISOR HYPERVISOR

VIRTUAL DISTRIBUTED SWITCH

WEB PORT GROUP

APP PORT GROUP

DB PORT GROUP

vCenter Server / vShield

8

5

1

9 ACI Fabric

Automatically Map EPG To Port Groups

Push Policy

Create VDS 2

Cisco APIC and VMware vCenter Initial

Handshake

6

DB DB

7 Create Port Groups

ACI Hypervisor Integration – VMware DVS/vShield

APIC

3

Attach Hypervisor to VDS

4 Learn location of ESX Host through LLDP

130


Southbound OpFlex API

VM VM VM VM

N1KV VEM

vSphere

Hypervisor Manager

§  OpFlex Control protocol -  Control channel -  VM attach/detach, link state

notifications §  VEM extension to the fabric §  vSphere 5.0 and above §  BPDU Filter/BPDU Guard §  SPAN/ERSPAN §  Port level stats collection §  Remote Virtual Leaf Support

(future)

Application Virtual Switch (AVS) Integration Overview

131


APIC Admin

VI/Server Admin Instantiate VMs, Assign to Port Groups

L/B

EPG APP

EPG DB F/W

EPG WEB



Web Web Web App


Application Virtual Switch (AVS)

WEB PORT GROUP

APP PORT GROUP

DB PORT GROUP

vCenter Server

8

5

1

9 ACI Fabric

Automatically Map EPG To Port Groups

Push Policy

Create AVS VDS 2

Cisco APIC and VMware vCenter Initial

Handshake

6

DB DB

7 Create Port Groups

ACI Hypervisor Integration – AVS

APIC

3

Attach Hypervisor to VDS

4 Learn location of ESX Host through OpFlex

OpFlex Agent OpFlex Agent

132


VM Attribute EPG Classification with AVS 11.1


End-Points end EPG membership

Server

Virtual Machines & Containers

Storage

Client 134

•  Endpoint == Workload unit connected to network directly or indirectly

•  An endpoint has address (identity), location, attributes (version, patch level)

•  Can be physical or virtual or container •  End Point Group (EPG) membership defined by:

•  Ingress physical port (Leaf or FEX) •  Ingress logical port (VM port group) •  VLAN ID •  VXLAN (VNID) •  IP Prefix/Subnet (so far only applicable to external/border

leaf connectivity) •  VM-based attributes (11.1 release) •  IP address (planned for 11.1(MR2) – Sept 2015)


Hypervisor Integration with ACI 11.0 EPG Classification via Port Groups

•  VM’s are placed within the Port Group defined for each EPG •  Traffic is encapsulated with the specific VLAN or VXLAN assigned to that port group

on that port and forwarded upstream to the TOR

VXLAN VNID = 5789

VXLAN VNID = 11348

802.1Q VLAN 50

Payload IP GBP VXLAN VTEP

VXLAN Leaf VTEP

802.1Q vSwitch

WEB PORT GROUP

APP PORT GROUP

vSwitch

WEB PORT GROUP

APP PORT GROUP

802.1Q VLAN 125

Payload IP Payload IP

Port Groups Created for Each EPG


Hypervisor Integration with ACI EPG Classification via VM Attributes

•  End Point Groups (EPG’s) can leverage multiple methods to ‘classify’ an endpoint or traffic from an endpoint

•  VM Port Groups Provide a simple mechanism to correlate a VM to a specific policy group

•  VM Attributes can also be used to classify a VM as a member of an EPG

•  Leverage ACI release 11.1 with AVS (initial deployment)

•  Support for other Hypervisor switches VMware vDS, Microsoft vSwitch, OVS (future)

VM Attribute Guest OS

VM Name

VM (id)

VNIC (id)

Hypervisor

DVS port-group

DVS

Datacenter

Custom Attribute

MAC Address

IP Address

vCenter VM

Attributes

VM Traffic

Attributes


Hypervisor Integration with ACI EPG Classification via VM Attributes

•  There are two categories of Attributes supported with the 11.1 release

•  VM Attributes (set by server administrator on creation of the VM)

•  VM Traffic Attributes (VM MAC/IP address or L4 port being used by the application)

•  Any endpoint placed within a Port Group on the vSwitch can be micro-classified based on the specific VM Attributes

•  Dynamic classification or re-classification •  e.g. Re-classify an endpoint that has been

detected to have a security exposure (move to quarantine security group)

VM Attribute Guest OS

VM Name

VM (id)

VNIC (id)

Hypervisor

DVS port-group

DVS

Datacenter

Custom Attribute

MAC Address

IP Address

vCenter VM

Attributes

VM Traffic

Attributes


AVS with ACI 11.1 EPG Classification via VM Attributes

vSwitch (AVS)

Port Group EPG == VM Attribute ‘x’

EPG == VM Attribute ‘y’

APIC Admin Create an EPG == VM Attribute ‘x’ on VMM Domain ‘A’

34 APIC Distributes VM

Attribute Policies to Leaf nodes

AVS notifies Leaf of VM Attach via

OpFlex Channel

6

Leaf Determines Attribute to EPG

Classification

7

Leaf Pushes EPG encapsulation

binding to AVS via OpFlex Channel

8

802.1Q VLAN 50

AVS forwards traffic with the correct EPG label (encapsulation)

9

APIC Retrieves Hypervisor State (VM State & VM

Attributes) & Initiate a Listener Process for any changes/

updates

2

Administrator Creates new vDS

(AVS)

1

VI/Server Admin

Boot new VM with desired VM Attributes

5


ACI Hypervisor Integration – Vmware vCenter View

139


VMware vCenter Plugin View

140



141



142


Microsoft SCVMM and Azure Pack Integration

Cisco Confidential 144

Microsoft Interaction with ACI Two modes of Operation

•  Policy Management: Through APIC •  Software / License: Windows Server with

HyperV, SCVMM •  VM Discovery: OpFlex •  Encapsulations: VLAN, VXLAN and

NVGRE (Future) •  Plugin Installation: Manual

Integration with SCVMM

APIC

Integration with Azure Pack

APIC

•  Superset of SCVMM •  Policy Management: Through APIC or

through Azure Pack •  Software / License: Windows Server with

HyperV, SCVMM, Azure Pack (free) •  VM Discovery: OpFlex •  Encapsulations: VLAN, VXLAN and

NVGRE (Future) •  Plugin Installation: Integrated

+

144


APIC Admin

SCVMM Admin Instantiate VMs, Assign to VM Networks

L/B

EPG APP

EPG DB F/W

EPG WEB



MSFT SCVMM

8

5

1

9 ACI Fabric

Automatically Map EPG To VM Networks

Push Policy

Create Virtual Switch

2

Cisco APIC and MSFT SCVMM Initial

Handshake

6

ACI and SCVMM Integration in 11.1 Release

APIC

3 Attach Hypervisor to Virtual Switch

4 Learn location of HyperV Host through OpFlex


OpFlex Agent

HYPERV VIRTUAL SWITCH

7 Create VM Networks

OpFlex Agent

WEB VM NETWORK

APP VM NETWORK

DB VM NETWORK

145

Web Web App App DB


APIC Admin (Basic Infrastructure)

Azure Pack Tenant

3

6

ACI Fabric

Push Network Profiles to APIC

Pull Policy on leaf where EP attaches

Indicate EP Attach to attached leaf when VM starts

1

2


ACI Azure Pack Integration in 11.1 Release

APIC

Get VLANs allocated for each EPG


7

Azure Pack \ SPF

SCVMM Plugin APIC Plugin OpFlex Agent OpFlex Agent OpFlex Agent

Instantiate VMs

5

1

4Create VM Networks

4

146

Web Web Web Web App App DB DB


Microsoft Azure Pack Integration §  Integration with Microsoft requires:

-  Windows Server 2012 -  Systems Center 2012 R2 with

SPF -  Windows Azure Pack

§  Azure Pack provides single pane of glass for Definition, creation, management of their cloud service

§  Divided into Provider (Admin) portal and Consumer Self-Service (Tenant) portal

§  Cisco ACI Service Plugin enables management of Network Infrastructure through APIC REST API

R2 w/ Service Provider Foundation

Web Sites

Service Plans Users

Provider Portal

Consumer Self-Service

Portal

Web Sites Apps Database VMs ACI

Service Provider Customer

VMs SQL Service Bus …

147


Cisco ACI Network Offerings Features Shared Network Virtual Private Network

Isolated Networks ✓ ✓

Firewall ✓ ✓

Shared DHCP ✓ ✓

Shared Load Balancer ✓ ✓

Shared Services ✓ ✓

Public Internet Access ✓ ✓

Private Address Space ✓

Private DHCP Server ✓


Use Cases Shared Network and Virtual Private Network

WEB

WEB

APP

APP

Finance Tenant

DB

MONGO DB

Shared Services Tenant

DHCP

DNS

ACI Common services

LB

FW

WEB

WEB

APP

APP

DevTest Tenant

192.168.0.0/16

APP APP

Finance Tenant

DHCP

DNS

ACI Common services

LB

FW

WEB WEB

APP APP

DevTest Tenant

192.168.0.0/16

WEB WEB WEB WEB DB

MONGO DB

Shared Services Tenant

10.0.10.0/24 10.0.10.0/24


Microsoft Azure Pack Integration Admin Experience

Add & Configure APIC, tenants, and VLAN ranges

Usage & Billing statistics per user and other admin functions

150

Role Based Access Control for Shared Services


Microsoft Azure Pack Integration Admin Experience

Network and Compute resources tenant has access to

Application Network Profiles are created through Azure Pack, and pushed to APIC using REST APIs

ACI constructs available to tenant

F5 or Citrix Load Balancer that is part of ACI Fabric

Shared Services


Microsoft Azure Pack Integration Tenant Experience

Network and Compute resources tenant has access to

Application Network Profiles are created through Azure Pack, and pushed to APIC using REST APIs

ACI constructs available to tenant


Openstack and KVM/OVS Integration


Why Cisco ACI and OpenStack

TELEMETRY AND OPERATIONS 5

•  Health Metrics •  Visibility •  Troubleshooting

•  Automation •  Intent-driven

GROUP-BASED POLICY SUPPPORT 1

•  Service chaining •  App Acceleration

SERVICE CHAINING 4 PHYSICAL +

VIRTUAL

•  Zero-touch Performance

•  Physical server •  Multi-hypervisor

2

•  Automatic VXLAN

•  Distributed L2 •  Distributed L3

FABRIC TUNNELS 3

•  Service chaining and redirection


Web

Web

Web

Web

App

App

DB

DB


NEUTRON ROUTER

SECURITY

GROUP

NEUTRON NETWORK

Contract Contract Contract

DB APP WEB ADC F/W

ADC

APIC Driver OVS Driver

Neutron Networking

Group Policy

OVS Driver

Neutron Networking

APIC Group Driver

Web

Web

Web

Web

App

App

DB

DB


Two Options for ACI APIC Driver (ML2) Group Policy Plugin


NEUTRON ROUTER

SECURITY

GROUP

Web

Web

Web

Web

App

App

DB

DB


NEUTRON NETWORK

APIC Driver OVS Driver

Neutron Networking

•  ML2 (modular level 2) driver supporting existing Neutron APIs: network, router, security group, LBaaS, etc.

•  Automation of neutron ports for virtual machines

•  Relies on OVS in hypervisor

•  Shipping today from Cisco

•  Available on Openstack IceHouse, Juno, etc.

APIC Driver for OpenStack APIC Driver (ML2)


APIC Driver Details

Neutron Workflow 1.  User creates a network / router / etc. through Neutron CLI / Horizon / Heat 2.  OVS Driver selects VLAN from VLAN pool. VLAN is configured in Open vSwitch 3.  APIC Driver maps neutron object to APIC policy model 4.  IP Tables in Linux Hypervisor provides host-based security group enforcement 5.  Open vSwitch tags each Neutron network with VLAN 6.  ACI ToR translates VLAN into VXLAN, providing distributed L2 and distributed default gateway support.

OVS Driver

Neutron Networking

APIC Driver

Hypervisor Hypervisor Hypervisor Hypervisor Hypervisor Hypervisor

ACI Fabric Offers: •  VXLAN tunnels •  Distributed L2 •  Distributed default

gateway

Hypervisor: •  Enforces security

groups


What’s Wrong with OpenStack Networking Today?

Service A

Service B Service C

Cloud Application Model Neutron Model

Network / subnet

Router External Network

Network / subnet

•  L2 / Broadcast is the base API! •  Network / routers / subnets •  Based on existing networking models •  No concept of dependency mapping or

intent

•  No broadcast / multicast •  Resilient / Fault Tolerant •  Scalable Tiers •  Built around loosely coupled services •  Don’t care about IP addresses


Where Can We Do Better

§  Build self-documenting dependency maps of tiers of an application

§  Define network service chains between tiers of an application without low level configuration

§  Separate application requirements from low level APIs

§  Separate tenant from operator

Separation of Concerns Enable Network Services

Dependency Mapping

Service A

Service C

Abstract Application API

Low level / Detailed API Service

A Service

C

Service A consumes service B and Service C

Service B

Service A

Service C

FIREWALL

Operator / Admin

OpenStack Tenant


Introducing Group-Based Policy •  Intent-based API for describing application requirements

•  Separates concerns of tenants and operators

•  Captures dependencies between tiers of an application

•  Plugin model •  Supports mapping to Neutron APIs •  Supports “native” SDN drivers

Policy Rules Set Web Group

Classifier Action

FIREWALL

DB Group

Classifier Action

Service Chain


OpenStack GBP Architecture

Neutron Driver maps GBP to existing Neutron API and offers compatibility with any existing Neutron Plugin

Native Drivers exist for OpenDaylight as well as multiple vendors (Cisco, Nuage Networks, and One Convergence)

Group Policy

CLI Horizon Heat

Neutron Driver

Neutron Any Existing Plugins and ML2 Drivers

Open model that is compatible with ANY physical or virtual networking backends

Native Driver 1

1

2

2


Group-Based Policy Model Policy Group: Set of endpoints with the same properties. Often a tier of an application.

Policy RuleSet: Set of Classifier / Actions describing how Policy Groups communicate.

Policy Classifier: Traffic filter including protocol, port and direction.

Policy Action: Behavior to take as a result of a match. Supported actions include “allow” and “redirect”

Service Chains: Set of ordered network services between Groups.

L2 Policy: Specifies the boundaries of a switching domain. Broadcast is an optional parameter

L3 Policy: An isolated address space containing L2 Policies / Subnets

L3 Policy

Policy Rule Set

Policy Rule Policy Rule

Service Chain

Classifier Action

Classifier Action

L2 Policy

Policy Group

Policy Target

Policy Target

Policy Target

Policy Group

Policy Target

Policy Target

Policy Target

L2 Policy

provide consume

Node Node


Contract Contract Contract

DB APP WEB ADC F/W

ADC

Group Policy

OVS Driver

Neutron Networking

APIC Group Driver

Web

Web

Web

Web

App

App

DB

DB


•  OpenStack extensions on top of Neutron exposing a policy API

•  Supports policy API to APIC •  Backwards compatible with existing neutron

plug-ins (works with Nexus 9000 standalone)

•  Available for Openstack Juno (Q1 CY 15)

•  Open approach

•  Enables Openstack customers to deploy, scale and modify policy across teams fast

Group-Based Policy APIC Driver (ML2)


Group Policy Plugin ACI Fabric Offers: •  VXLAN tunnels •  Distributed L2 •  Distributed default

gateway •  Security enforcement

Neutron Workflow 1.  User creates Group-Based Policy through CLI / Horizon / Heat. 2.  OVS Driver selects VLAN from VLAN pool. VLAN is configured in Open vSwitch 3.  APIC Driver maps GBP to APIC policy 4.  Non-OpFlex: All inter-EPG traffic sent to ToR for enforcement (note, with OpFlex switching and enforcement may occur in OVS). 5.  Open vSwitch tags each group with VLAN 6.  ACI ToR translates VLAN into VXLAN, providing distributed L2, security policy, and distributed default gateway support.

OVS Driver

Neutron Networking

APIC Group Driver

Hypervisor Hypervisor Hypervisor Hypervisor Hypervisor Hypervisor

Group Policy


Install and try GBP now! •  Available with OpenStack Juno release via StackForge

•  https://github.com/stackforge/group-based-policy

Runs with ML2 / OVS in a VM!

Try it now:

•  git clone http://github.com/group-policy/devstack -b juno-gbp

•  cd devstack;

•  stack.sh

Packaging and support available through Cisco and its partners Red Hat, Mirantis, Canonical in progress


OpenStack Partners

Support for major OpenStack Distributions

Testing and Integration Working closely with vendors to test and qualify APIC Plugin on

OpenStack distributions

Easy Deployment Integrating with existing

deployment tools used by each distribution

Customization to ACI Evaluating ways to expose features that ACI can leverage such as Group Policy

and OpFlex

For Your Reference


Support Matrix Vendor Distribution Deployment

ToolChain Base Operating System

Ubuntu OpenStack Juju Ubuntu 14.04

Red Hat OS 5 Foreman RHEL 7

Mirantis OpenStack 5 Fuel Ubuntu 12.04

Mirantis OpenStack 5 Fuel Centos 6.5

Mirantis 6 + RHEL OSP 6 testing in progress

For Your Reference


LINUX Container Integration


Hypervisors vs. Linux Containers

Hardware

Operating System

Hypervisor

Virtual Machine

Operating System

Bins / libs

App App

Virtual Machine

Operating System

Bins / libs

App App

Hardware

Hypervisor

Virtual Machine

Operating System

Bins / libs

App App

Virtual Machine

Operating System

Bins / libs

App App

Hardware

Operating System

Container

Bins / libs

App App

Container

Bins / libs

App App

Type 1 Hypervisor Type 2 Hypervisor Linux Containers (LXC)

Containers share the OS kernel of the host and thus are lightweight. However, each container must have the same OS kernel.

Containers are isolated, but share OS and, where appropriate, libs / bins.


Hypervisor VM vs. LXC vs. Docker containers


•  Open-Source Container for Dummies

•  Open Source engine to commoditize LXC

•  Create lightweight, portable, isolated, self-sufficient container from any application.

•  Delivers on full DevOps goal: •  Build once… run anywhere. •  Configure once… run anything

•  Ecosystems! OS, VM’s, PaaS, IaaS…

What is containers ?


SECURITY

Trusted Zone

DB Tier

DMZ

External Zone

APP DB WEB EXTERNAL ACI Policy

ACI Policy

ACI Policy

172

Abstracting / Mapping via ACI’s Application Network Profiles

! ! !FW

ADC

Virtual Machines Docker Containers Bare-Metal Server

172




SECURITY

Trusted Zone

DB Tier

DMZ

External Zone

APP DB WEB EXTERNAL ACI Policy

ACI Policy

ACI Policy

Option 1: Supporting Containers with ACI policy model via OpFlex on OVS

! ! !FW

ADC

Virtual Machines Docker Containers Bare-Metal Server


ACI Virtual Leaf: OpFlex + OVS


H1CY15


ACI Fabric

EPG A

EPG B

EPG = VLAN

ACI Contract 1)  Load the ACI Toolkit on your machine (documentation is at http://datacenter.github.io/acitoolkit/docsbuild/html/genindex.html)

2)  Run the Toolkit to automate the following:

1)  Create the ACI constructs: Tenant, BD, context, Application Network Profile, EPG, Contract

2) Attach physical interfaces to EPG(s)

3) Create a VLAN interface:

4) Attach the logical interface (VLAN) to the Physical Interface 5) Attach the EPG to the logical interface

Option 2: Supporting Containers with ACI policy model via MACVLAN on Linux


ACI Fabric

! !! ! ! !! ! ! ! !

20

20

30

30

EPG A

EPG B

EPG = VLAN

ACI Contract

3)  Example with LXC

# Show the EPGs on the APIC aci-show-epgs.py # Create the container lxc-create --template ubuntu --name container_name # Attach the container to the EPG aci-attach-epg.py --container container_name --epg epg_name # Start the container lxc-start --name container_name

4)  Example with Docker “docker run” with “macvlan” network type •  allows to map the docker container (MAC) to a VLAN by the “fire up” of

the Docker container •  VLAN got previously mapped to EPG via interface (physical or trunk) •  Connectivity is done without “virtual switching” which increases

performance •  cross-server / cross-racks policy consistency granted via ACI.

•  P.S.: you may consider to previously run a network type “empty” to remove the masquerade rule and not have the default docker0 associated with br0 linux bridge

Option 2: Supporting Containers with ACI policy model via MACVLAN on Linux

Cisco Confidential 176 © 2014 Cisco and/or its affiliates. All rights reserved.

ACI Fabric – DC 01 ACI Fabric – DC 02

Docker-based Web Application Docker-based Web Application

ACI Application Network Profile

Data Center 01 Data Center 02

Multi-site abstraction and portability of Network Metadata and Docker-based Applications


http://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/application-centric-infrastructure/white-paper-c11-732697.html

Docker and ACI


ACI Integration of Layer 4 – 7 Services


What is NOT Simple Today? Challenges with Network Service Insertion

Router

Router

Switch

LB

FW

vFW

servers

1.  Configure Network to Insert Firewall 2.  Configure firewall network parameters 3.  Configure firewall rules as required by the

application 4.  Configure Load Balancer Network

Parameters 5.  Configure Router to steer traffic to/from Load

Balancer 6.  Configure Load Balancer as required by the

application


Intended design

Physical server Virtual Server

I want virtual firewalling in between with ASA version a.b

I want physical firewalling in between with F5 version a.b and Firewall version c.d.

180


Automate Service Insertion Through APIC

APP DB WEB EXTERNAL

APIC Policy Model Endpoint Group (EPG): Collection of similar End Points identifying a particular Application Tier. Endpoint could represent VMs, VNICs , IP, DNS name etc

Application Profile: Collection of Endpoint Groups and the policies that define way Endpoint group communicate with each other

Application profile

Policy Policy Policy

181


ACI Service Insertion via Policy •  Automated and scalable L4-

L7 service insertion

•  Packet match on a redirection rule sends the packet into a services graph.

•  Service Graph can be one or more service nodes pre-defined in a series.

•  Service graph simplifies and scales service operations

Begin End Stage 1

FW_A

DC

1

EPG 2

EPG 1

Application Admin

Service Admin

ASA 5585

Netscaler VPX

Chain “FW_ADC 1”

Policy-based Redirection

Stage 2


Intended Design Goal

Default Gateway Transparent firewall with virtual ASA


Create Service Graph

184


Associate Graph to a Contract

185


APIC

L4-7 Plugin API (Device Package) •  APIC interfaces with the device using

python scripts

•  APIC calls device specific python script function on various events

•  APIC uses device configuration model provided in the device package to pass appropriate configuration to the device scripts

•  Device script handlers interface with the device using its REST or CLI interface

•  Open Specification

Device Spec (XML)

Device Script (Python / CLI)

Uses Device’s native API

186


Device Package Example

Following functions can be configured through APIC 187


Configure Function Parameters

188


Bridge Domain Outside Bridge Domain Inside

L3Out L3InstP

Server EPG

service graph

Contract Provider Consumer

VRF This is just to make the Policy model happy

ARP flooding unicast flooding no ip routing

subnet, i.e. default gateway for servers hardware proxy

Service Graph with the Policy Model


Service Configuration before the Service Graph

192.168.1.1 192.168.1.100

10.1.1.1

172.16.1.1

192.168.100.1

HTTP (TCP/80) HTTPS (TCP/443) DCERPC (TCP/135) SSH (TCP/22) ICMP

access-list OUT permit tcp host 192.168.1.1 host 10.1.1.1 eq 80 access-list OUT permit tcp host 192.179.1.1 host 10.1.1.1 eq 443 […] access-list OUT permit icmp host 192.168.1.100 host 192.168.100.1

30 ACL Rules

172.18.20.13

access-list OUT permit tcp host 172.18.20.13 host 10.1.1.1 eq 80 access-list OUT permit tcp host 172.18.20.13 host 10.1.1.1 eq 443 […] access-list OUT permit icmp host 172.18.20.13 host 192.168.100.1

15 ACL Rules

45 ACL Rules

Network Admin Security Admin

Add client 172.18.20.13, call Security Admin to

enable access

Remove client 192.168.1.1, “no other action necessary”

Add ASA rules for client 172.18.20.13

Original ASA rules never change 4

1

2

2

3

4

Files

Users


Automatic endpoint addition/removal with ACI

10.1.1.1

172.16.1.1

192.168.100.1

Servers

192.168.1.1

192.168.1.100

172.18.20.13 HTTP (TCP/80) HTTPS (TCP/443) DCERPC (TCP/135) SSH (TCP/22) ICMP

Source EPG

Leaf 1, port 1 Users


Destination EPG

Leaf 3, port 2 Servers




Network Admin

Add client 172.18.20.13, use existing ASA instance

Remove client 192.168.1.1

Security Admin Insert ASA instance in the service graph with desired policies

Same 5 service rules and actions

ASA1

Clients

Port Rules

access-list OUT permit tcp any any eq 80 access-list OUT permit tcp any any eq 443 access-list OUT permit tcp any any eq 135 access-list OUT permit tcp any any eq 22 access-list OUT permit icmp any any


FirePOWER in ACI


Advanced Threat Protection with FirePOWER + ACI

FireSIGHT Management Center

Alerts Network Visibility Policy Management Analytics Remediation

•  Situation –  Advanced threats that are not detected by

conventional security products –  Limited security resources

•  ACI Solution –  Automated provisioning of NGIPS and

Advanced Malware Protection –  Visibility and awareness with FireSIGHT –  Continuous analysis –  Physical and virtual appliances

•  Benefits –  Industry-leading security efficacy –  Automation and correlation for reduced TCO –  Retrospective security helps scope, contain

and remediate

Automated Feedback Loop for Intelligent Threat Response

WEB

WEB WEB WEB

DB

DB DB DB

APP

APP APP APP AMP NGIPS

AMP NGIPS


Preserve Separation of Duties

SecOps

DevOps/Network Admin


Configuration Model

Device Interface: REST/CLI

APIC Script Interface Python Scripts

Script Engine

APIC– Policy Manager

Physical Virtual


EPG “Internet”

EPG “Web”

FirePOWER Services For ACI – Intelligent Threat Defense


Alerts Network Visibility Policy Management Analytics Remediation

Application Policy Infrastructure

Controller (APIC)

Service Graph Contracts

NGIPS/NGFW Advanced Malware Protection

Policy and events

Basic configuration and health

Intelligent Remediation


UNT PUBLIC

Trusted – No Graph CORP

APIC 172.28.199.30

Move IP to Quarantine

Defense Center

10.0.0.244

FW NGIPS 10.1.0.234

Relaxed

REST calls to APIC NB API

ACI Fabric

N9K Leaf Switch

FirePOWER Appliance 10.0.1.30

SPAN Traffic

Attack ESXi – 10.1.0.44

1.1.1.6 1.1.1.7

FW QUA

Strict

REM

1.1.1.3

Security Feedback Loop


§  Cisco® ASAv running Release 9.2(1) and later and Cisco ASA 5585-X running Release 8.4(1) and later §  Cisco ASA Release 9.2(2) and later is recommended for all appliances

§  Device specification §  Hierarchical model of the device capabilities in Cisco APIC

§  E.g., the list of supported features that are configurable by the Cisco APIC user

§  Function-independent vs. function-specific parameters

§  Device script §  Converts Cisco APIC specific API function calls into Cisco ASA CLI script over HTTPS

§  E.g., how to configure an ACL or interface on Cisco ASA with the given parameters from Cisco APIC

§  Add/delete/modify or monitor health


Routed Mode

Transparent Mode

External EPG E1

App-A EPG FW

Graph A 10.0.0.0/24

10.0.0.1 20.0.0.1 20.0.0.0/24

Tenant A

Consumer Provider

EPG A EPG B FW

Graph A

10.0.0.0/24

Consumer Provider

Tenant A


•  Routed Mode

•  Transparent Mode

EPG A EPG B FW

10.0.0.0/24

Tenant A External Internal

EPG A EPG A FW

10.0.0.1 20.0.0.1

Tenant A External Internal

VRF VRF

OSPF/BGP

OSPF/BGP OSPF/BGP

VRF VRF

10.0.0.2 20.0.0.2

10.0.0.10 10.0.0.11 100.0.0.0/24 200.0.0.0/24

201.0.0.0/24

202.0.0.0/24

203.0.0.0/24

101.0.0.0/24

102.0.0.0/24

103.0.0.0/24

200.0.0.0/24

201.0.0.0/24

202.0.0.0/24

203.0.0.0/24

100.0.0.0/24

101.0.0.0/24

102.0.0.0/24

103.0.0.0/24


Cisco® ACI Fabric

Cisco ASA Cluster Flow Symmetry Within Service Graph

Stateless Load Balancing

Stateful Flow Asymmetry on Changes

Elastic Scalability

Asymmetry Compensation


Cisco Security + ACI Roadmap

ASA, FP, NGFW = EC/AC = CC/BC = Roadmap

Q2CY15 4QCY15

Release & Commit Status FCS+9 (ACI 11.1) FCS+12 - ACI 11.1(1)

ASA •  Support for Multi-context •  Support for BGP •  Support for OSPF support •  Support for ASA + FirePOWER

Services (5585)

•  Support for SGACL/SXP configuration

•  Support for S2S VPN •  Support for RAVPN

FirePOWER •  Device Package 1.0 •  FirePOWER Threat Capabilities •  Switched interfaces

•  Usability Enhancements •  Add missing management

functions

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 201 Cisco Confidential – Redistribution Prohibited


ACI L4-L7 – Device Package Update Device Package ETA

F5 (Big IP physical and virtual) Now ASA (5585 8.4 and ASAv 9.2.1) Now Citrix (NetScaler MPX, SDX, VPX, NetScaler 1000v) Now A10 Now Radware ADC Now Avi Networks Now Cisco Sourcefire Q2 CY15 Fortinet Q2 CY15 Palo Alto Networks Q2 CY15 Check Point Q3 CY15 Radware DefensePro Q3 CY15 Intel Security - McAfee Q3 CY15 Symantec Data Loss Prevention Q3 CY15


Programmability and ACI


Virtual Machines

LXC / Docker Containers

Apps Portability, Cross-Platform & Automation

Applications PaaS

Two Market Transitions – One DC Network

Traditional Data Center Networking

Network

Apps Policy

Application Centric Infrastructure (ACI)

Network + Services Abstraction & Automation

Infrastructure HyperScale Data Centers

DC Switching


We currently have :

•  REST API

•  Full Object Model exposed

•  JSON or XML

•  Python SDK for accessing object model

PROGRAMMABILITY & ACI


DB APP ADC WEB F/W

ADC

Typical Application Network Profile on ACI


EPG 100 EPG 200

App 1 App 2

10.10.40/24

10.10.30/24

10.10.20/24

10.10.10/24

VLAN 400 10.10.40/24

VLAN 300 10.10.30/24

VLAN 200 10.10.20/24

VLAN 100 10.10.10/24

Apps Coupled to Location

ACL-based Policy Per Interface

Visibility At Network or VLAN Level

No Address Independence or Policy Mobility

Apps Decoupled from Location

Visibility At App or Group Level

Policy Between Groups

Complete Address Independence & Policy

Mobility

Traditional Network Model Application Centric Infrastructure

EPG 100

EPG 200

EPG 300

EPG 400

EPGs @ ACI bring true network abstraction, as needed

207


From Development to Test to Production

EPG Dev

DEV DEV

EPG Test

TEST TEST

EPG Prod

PROD PROD

Development lifecycle push as code progresses EPGs can be used to segregate separate development phases.

208


Leveraging Declarative Modeling for Application Profiles

WAN

Firewall

LB to EPG 2

Connect to EPG 3

Connect to EPG 2

High Priority

EPG1 EPG2 EPG3

APPLICATION PROFILE

Security Governance Service Level Scalability Availability Performance

ADC F/W ADC

WEB APP DB


WEB APP Database Load Balancer

User/Client Browser

Example of EPG allocation and associated ACI contracts on a 3-Tier video application

External EPG Front-End-Scale EPG Web EPG APP EPG DB EPG


On-going App Development evolution towards Cloud model

From Traditional Monolithic Multi-tier App to Cloud-Aware App


Load Balancer

Client

Product Info Service

Order Service

Feedback Loop

Management

Cache-Fill

Cache Control

Streaming

OLTP

OLAP

Real Time

Historical

REST

REST

Thrift

API Gateway

Rest Proxy

Event Publishing

Browser

REST Client

Content Router

Product Info UI

Order Service UI

Feedback Loop UI

Service Registry

Load Balancer

Same video application example as microservices-based Cloud-App


Load Balancer

Client

Product Info Service

Order Service

Feedback Loop

Management

Cache-Fill

Cache Control

Streaming

OLTP

OLAP

Real Time

Historical

API Gateway

Rest Proxy

Event Publishing

Browser

REST Client

Content Router

Product Info UI

Order Service UI

Feedback Loop UI

Service Registry

Load Balancer

Potential ACI EPG and contracts allocation on a Cloud-App


We currently have :

•  REST API

•  Full Object Model exposed

•  JSON or XML

•  Python SDK for accessing object model

But….

•  Steep learning curve •  5000+ classes •  New concepts, etc.

PROGRAMMABILITY & ACI


•  Ease the learning curve

•  Remove some initial frustration

•  Address 80% of the use cases

•  Provide examples and sample scripts for customers

•  Accelerate ACI adoption

ACI TOOLKIT – GOALS


Cisco ACI Toolkit

Infrastructure as Code

https://github.com/datacenter/acitoolkit http://datacenter.github.io/acitoolkit/


•  Simple toolkit built on top of APIC API •  Set of simple python classes

Python Library Used to generate REST API calls Runs locally

•  Small number of classes ~30 currently “Intuitive” names

•  Not full functionality, most common Focused primarily on configuration

•  Preserves the ACI basic concepts Tenants, EPGs, Contracts, etc.

•  Expose ACI to DevOps as a library / code

APIC

ACI Toolkit

Linux Commands

NX-OS like CLI

Do It Yourself

Cisco ACI Toolkit


Tenant

Context / VRF

Subnet

Bridge Domain

Outside EPG

EPG

App. Profile

Taboo

Contract

Filter Entry

L3 Interface

L2 Interface

Interface

Node

Link

POD

1

* * 1 * * 1

1

* *

* * *

*

Provide / Consume

1

* *

*

*

*

1 1

* *

1

*

Provide / Consume

*

*

1

1

* 1

1

1 1

1

1 * 1 *

1

2

1

*

1

*

1

*

Network Physical Policy ACI Toolkit Policy Model

Jan/2015


ACI Endpoint Tracker Application •  Tracks all attachment, detachment,

movement of Endpoints in ACI fabric

•  Stores activity in open source MySQL Database, allowing query capabilities

•  Provides foundation for visualization and query tools

•  Some questions that could be solved: •  What are all the Endpoints on network? •  Where is a specific Endpoint? •  What was connected last Thursday

between 3:30am and 4:00am? •  What is the history of a given Endpoint?


http://goo.gl/agx8gZ

Docker and ACI

https://registry.hub.docker.com/u/dockercisco/aci/


ACI + UCS are unique as the foundation for the App market transition

ACI + UCS

Traditional Monolithic Multi-tier App Cloud-Aware App

ACI supports physical, virtual and container based workloads as well as API and code / library based consumption. Allows business to change app models at their speed.


GitHub – a resource for ACI scripts and tools •  ACI Toolkit:

http://datacenter.github.io/acitoolkit/ https://github.com/datacenter/acitoolkit

•  ACI Diagram https://github.com/cgascoig/aci-diagram

•  ACI Endpoint Tracker http://datacenter.github.io/acitoolkit/docsbuild/html/endpointtracker.html

225


ACI Toolkit – using the APIC API’s •  Fulfill prerequisites:

Python 2.7+ setuptools package (apt-get install python-pip – installs setuptools too) requests library (pip install requests) websocket-client library (pip install websocket-client)

•  Get acitoolkit: git clone https://github.com/datacenter/acitoolkit.git

•  Install acitoolkit: cd acitoolkit python setup.py install 226

application centric infrastructure (aci), the policy driven data centre

Technology