h20628. · 2019-01-17 · these vlans are configured on this l3 switch as svi for its network ip...

http://h17007.www1.hpe.com/docs/enterprise/servers/oneview2.0/cic/en/content/index.html

http://h17007.www1.hpe.com/docs/enterprise/servers/oneview2.0/cic/en/content/index.html

https://h20392.www2.hpe.com/portal/swdepot/displayProductInfo.do?productNumber=Z7550-63180

http://h17007.www1.hpe.com/us/en/enterprise/converged-infrastructure/info-library/index.aspx?&type=5+11

https://www.youtube.com/watch?v=juYvy99v2dQ&feature=youtu.be

Mgmt-B,

https://www.hpe.com/h20195/V2/GetPDF.aspx/4AA5-0351ENW.pdf

http://ow.ly/UzSDz

vMotion-A vMotion-B

VMM-Pool-B_

Connection-B

-Set-A

Uplink-Set-A

Uplink-Set-B

ACI-DC

Standard Aggregation L3 Switch Configuration for “existing Layer 2”:Nexus 6K-01 switch is used as upstream Agg L3 switch to provide gateway addresses for all the EPG VLANs used in this solution and routed. These VLANs are configured on this L3 switch as SVI for its network IP segment with HSRP and interface-vlan feature enabled to provide default gateway and routing. As a result, all network reachability between EPGs within ACI fabric or EPG to external devices outside ACI fabric must be routed at Agg L3 switch. (NX-6K-01). The following Fig 23 is the diagram for leaf node and upstream device connection. A detail switch configuration relates to this solution can be found in Appendix A

http://www.cisco.com/c/en/us/products/switches/application-virtual-switch/index.html

SOLUTION SPECIFIC DESIGN CONSIDERATIONSThe above diagram (Fig 24) shows the connections related to ACI vSphere VMM integration. The solution is based on a vSphere

vCenter VMM domain integration with the VMware virtual environment and extend the EPG out of the ACI Fabric to traditional

network layer 2 environment (where VLANs are used). In this design, ACI dynamically assigns VLAN ID from the pool to the EPG

portgroup and rely on the LLDP tagging feature enabled in HPE OneView for the VCs to determine the location of the workload VMs

so that the policies can be applied and enforced. ACI statically assign a leaf port (along with a VLAN ID) to an EPG. (Static Binding in

ACI EPG configuration and will be discussed in later “ s” section) so all the traffic

received on the leaf port with the configured VLAN ID will be mapped to the EPG and the configured policies for the EPG can be

enforced.

Since this design the VLANs’ Gateways address lives outside ACI Fabric (vs. Cisco ACI subnets & define pervasive gateway inside

fabric), there are some limitations, design considerations, and configuration requirements:

1. Inter – VLAN routing takes place outside the ACI fabric. This simulates a “brownfield” ACI implementation scenario

and means that ACI contracts can’t control the network traffic between VLANs.

2. Each EPG is created for each VLAN that is outside of the ACI fabric

3. Each EPG must be associated to the physical domain that contains the VLAN IDs used outside of ACI

4. Each EPG should be in its own dedicated Bridge Domain

5. The Bridge Domain flooding feature must be enabled to broadcast traffic to the external gateways.

ACI CONFIGURATION STEPS

http://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/application-centric-infrastructure/white-paper-c11-731960.html

L121-P52-vPC3-SP4

L1-P3-PG

Enable_CDP

L1-P3-VPool L1-P3-PD L1-P3-

VPool L1-P3-AEP

OV-DVS

OV-VC

ACI-DC

OV-DVS

ACI-DC ACI-DC”

eth 1/3

L1-P3-SP L1-P3-IP L1-

P3-PG L1-P3-AEP

L1-P3-AEP

Leaf node 1# show int brie --------------------------------------------------------------------------------

Port VRF Status IP Address Speed MTU

--------------------------------------------------------------------------------

mgmt0 -- up 10.16.42.104 1000 9000

--------------------------------------------------------------------------------------

Ethernet VLAN Type Mode Status Reason Speed Port ch#

--------------------------------------------------------------------------------------

Eth1/1 20 eth trunk up none 10G(D) --



OV-VMM-AEP

L12-P5-vPC-PG

L12-P5-vPC-SP L12-P5-vPC-IP

L12-P5-vPC-PG

OV-VMM-AEP L12-P5-vPC-PG L12-P6-vPC-PG

OV-VMM-AEP e1/5 & 6

Leaf node1# show interface brief

--------------------------------------------------------------------------------


--------------------------------------------------------------------------------

mgmt0 -- up 10.16.42.104 1000 9000

--------------------------------------------------------------------------------

Ethernet VLAN Type Mode Status Reason Speed Port

Interface Ch #

--------------------------------------------------------------------------------

Eth1/5 0 eth trunk up none 10G(D) 7


Leaf node 2# show inter brie

--------------------------------------------------------------------------------


--------------------------------------------------------------------------------

mgmt0 -- up 10.16.42.103 1000 9000

--------------------------------------------------------------------------------

Ethernet VLAN Type Mode Status Reason Speed Port

Interface Ch #

--------------------------------------------------------------------------------



HPE

HPE-VRF

App-BD

NETWORK TRAFFIC ENDPOINTS GROUPS (EPGS) BRIDGE DOMAIN (BD) VLAN (TRADITIONAL NETWORK)

10.16.140.0/24 Ext-mgmt Ext-mgmt - BD VLAN 140

10.16.141.0/24 vMotion vMotion - BD VLAN 141

10.16.143.0/24 Web Web - BD VLAN 143

10.16.144.0/24 App App - BD VLAN 144

10.16.145.0/24 DB DB - BD VLAN 145

http://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/sw/migration_guides/migrating_existing_networks_to_aci.html


http://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/application-centric-infrastructure/white-paper-c11-731960.html?referring_site=RE&pos=1&page=http://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/application-centric-infrastructure/guide-c07-736077.html

HPE HPE-ANP

HPE

syncs with vCenter to create a matching DVS portgroup.

Ext-mgmt vMotion

Web App DB

Ext-mgmt”

Ext-mgmt

Ext-mgmt

EPG-VLAN

C:\Users>ping 10.16.140.13

Pinging 10.16.140.13 with 32 bytes of data:

Reply from 10.16.140.13: bytes=32 time=110ms TTL=61


App Web DB) App

App

HPE|HPE-ANP-App

Web DB

DB,

HPE|HPE-ANP-xxx

HPE|HPE-ANP-App.

HPE|HPE-ANP-App

HPE|HPE-ANP-App

VM – App1 HPE|HPE-ANP-App

EPG VLAN

VM App1

VM App1

DIA-N6K-01#

10.16.144.141 00:00:10 0050.5689.21ed Vlan144

DIA-N6K-01# show mac address-table | in 21ed

* 144 0050.5689.21ed dynamic 20 Eth1/3

C:\Users\Administrator>






Ping statistics for 10.16.144.141:

Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),

Approximate round trip times in milli-seconds:

Minimum = 37ms, Maximum = 381ms, Average = 150ms

Following are a list of references and links.

https://h20392.www2.hpe.com/portal/swdepot/displayProductInfo.do?productNumber=Z7550-63180

https://www.hpe.com/h20195/v2/default.aspx?cc=us&lc=en&oid=5410258

http://h20628.www2.hp.com/km-ext/kmcsdirect/emr_na-c04731869-1.pdf

https://www.youtube.com/watch?v=juYvy99v2dQ

http://h20564.www2.hpe.com/hpsc/doc/public/display?docId=c04770680

http://h20564.www2.hpe.com/hpsc/doc/public/display?docId=c04770680

https://hongjunma.wordpress.com/

https://www.vmware.com/support/pubs/vsphere-esxi-vcenter-server-6-pubs.html

http://www.cisco.com/c/en/us/support/cloud-systems-management/application-policy-infrastructure-controller-apic/tsd-products-support-series-home.html

http://h20628.www2.hp.com/km-ext/kmcsdirect/emr_na-c04731869-1.pdf


http://www.cisco.com/c/en/us/products/collateral/cloud-systems-management/aci-fabric-controller/white-paper-c11-729906.pdf

CONFIGURATION ITEMS CONFIGURATION NOTES

VLAN Pool (VMM) OV-VMM-Vpool The dynamical VLAN pool for VMM domain. VLAN 1041-1050

VMM Domains OV-DVS Name of VMM domain is name of VDS in vCenter

Associated AEP OV-VMM-AEPOV_VMM-AEP is created during domain creation, associated with an

interface in interface policy group creation or modification.

Interface Profile

L12-P5-vPC-IP

L12-P6-vPC-IP

L12-P5-vPC-IP – leaf 1/2 .e1/5 – associated with L12-P5-vPC-SP.

L12-P6-vPC-IP – leaf 1/2, e1/6 - associated with L12-P5-vPC-SP

Interface policy group

L12-P5-vPC-PG

L12-P6-vPC-PG

define interface policy and associate AEP for interface (s), associated to

“OV-VMM-AEP” to enable VMM traffic

Switch Profiles

L12 – Leaf node 1 & 2, P# - refer to the port position on a switch the type of connection – Access port, PO or vPC.

EPGs (ACI)

Leaf node 1

Internal VLAN

Leaf Node 2

Internal VLAN

VxLAN ID Static Enc VLAN VMM Domain Agg.SW (Ext.) Interfaces

EPG-Ext-mgmt VLAN 50 VLAN 18 VLAN 140 N/A VLAN 140 E1/3, E1/5-6, Po7-8

EPG-Ext-mgmt BD VLAN 49 VLAN 17 15695751 E1/3, E1/5-6, Po7-8

vMotion VLAN 23 VLAN 56 VLAN 141 N/A VLAN 141 E1/3,E1/5-6, Po7-8

vMotion-BD VLAN 43 VLAB 46 15695750 E1/3, E1/5-6, Po7-8

Web (VMM) VLAN 17 VLAN 36 VLAN 143 VLAN 1050 E1/5-6, Po7-8

Web-BD VLAN 47 VLAN 35 15400874 E1/3, E1/5-6, Po7-8

Web (PD) VLAN 48 VLAN 143 VLAN 143 E1/3

App (VMM) VLAN 15 VLAN 38 VLAN 144 VLAN 1044 E1/5-6, Po7-8

App-BD VLAN 41 VLAN 37 15564692 E1/3, E1/5-6, Po7-8

App-(PD) VLAN 42 VLAN 144 VLAN 144 E1/3

DB (VMM) VLAN 18 VLAN 40 VLAN 145 VLAN 1041 E1/5-6, Po7-8

EPGs (ACI)

Leaf node 1

Internal VLAN

Leaf Node 2

Internal VLAN

VxLAN ID Static Enc VLAN VMM Domain Agg.SW (Ext.) Interfaces

DB-BD VLAN 45 VLAN 39 14811120 E1/3, E1/5-6, Po7-8

DB (PD) VLAN 46 VLAN 145 VLAN 145 E1/3

Agg-L3-SW# show feature | inc enabled

hsrp_engine 1 enabled

interface-vlan 1 enabled

lacp 1 enabled

lldp 1 enabled

vpc 1 enabled

vlan 140

name 10.16.140.0n24-VLAN140

vlan 141

name 10.16.141.0n24-VLAN141

vlan 142

name 10.16.142.0n24-VLAN142

vlan 143

name 10.16.143.0n24-VLAN143

vlan 144

name 10.16.144.0n24-VLAN144

vlan 145

name 10.16.145.0n24-Vlan145

interface Vlan140

no shutdown

no ip redirects

ip address 10.16.140.253/24

ip ospf passive-interface

ip router ospf core area 0.0.0.0

hsrp 140

preempt

priority 110

ip 10.16.140.1

ip dhcp relay address 10.16.140.252

interface Vlan141

no shutdown

no ip redirects

ip address 10.16.141.253/24


hsrp 141

preempt

priority 110

ip 10.16.141.1

interface Vlan142

no shutdown

no ip redirects

ip address 10.16.142.253/24


hsrp 142

preempt

priority 110

ip 10.16.142.1

interface Vlan143

no shutdown

no ip redirects

ip address 10.16.143.253/24



hsrp 143

preempt

priority 110

ip 10.16.143.1


interface Vlan144

no shutdown

no ip redirects

ip address 10.16.144.253/24



hsrp 144

preempt

priority 110

ip 10.16.144.1


interface Vlan145

no shutdown

no ip redirects

ip address 10.16.145.253/24



hsrp 145

preempt

priority 110

ip 10.16.145.1


interface Ethernet1/3

description To ACI-DIA-9396-01-P3 (Leaf1)

switchport mode trunk

…

Leaf1# show endpoint detail +-----------------+---------------+-----------------+--------------

VLAN/ Encap MAC Address MAC/ Interface Endpoint Group

Domain VLAN IP Address IP Info Info

+-----------------------------------+---------------+-----------------+--------------

42 vlan-144 002a.6aad.167c L eth1/3 HPE:HPE-ANP:App

HPE:HPE-VRF vlan-144 10.16.144.254 L eth1/3

42 vlan-144 002a.6aad.163c L eth1/3 HPE:HPE-ANP:App


42 vlan-144 0000.0c07.ac90 L eth1/3 HPE:HPE-ANP:App


15 vlan-1044 0050.5689.d45b LV po7 HPE:HPE-ANP:App

HPE:HPE-VRF vlan-1044 10.16.144.142 LV po7

15 vlan-1044 0050.5689.21ed LV po8 HPE:HPE-ANP:App



18 vlan-1041 0050.5689.b1c6 LV po8 HPE:HPE-ANP:DB


48 vlan-143 002a.6aad.167c L eth1/3 HPE:HPE-ANP:Web


48 vlan-143 002a.6aad.163c L eth1/3 HPE:HPE-ANP:Web


48 vlan-143 0000.0c07.ac8f L eth1/3 HPE:HPE-ANP:Web


…

Leaf2# show endpoint detail

+-----------------------------------+---------------+-----------------+--------------

VLAN/ Encap MAC Address MAC/ Interface Endpoint Group

Domain VLAN IP Address IP Info Info

+-----------------------------------+---------------+-----------------+--------------

36 vlan-1050 0050.5689.239a LV po7 HPE:HPE-ANP:Web


38 vlan-1044 0050.5689.d45b LV po7 HPE:HPE-ANP:App


38 vlan-1044 0050.5689.21ed LV po8 HPE:HPE-ANP:App


40 vlan-1041 0050.5689.b1c6 LV po8 HPE:HPE-ANP:DB


overlay-1 10.0.72.95 L lo0

overlay-1 10.0.16.64 L lo2

…

From the 1st column, “vlan domain”, the VLAN ID is the ACI system internal vlan ID. Each internal VLAN ID maps to a data VLAN (Encap VLAN in 2nd column) and “endpoint” belong to an EPG. In this example, an endpoint device is a VM with MAC address 0050.5689.21ed at IP address 10.16.144.141. It belong to “HPE:HPE-ANP:App” EPG and assigned VLAN 1044 as it’s VLAN ID within

ACI VMM domain. This VM data traffic is carried by leaf internal VLAN 15 on leaf node 1 and VLAN 38 on leaf node 2 on interface port channel 8. (two members in the Interface port channel 8: interface e1/6 on leaf node 1 and leaf node 2) In addition, Internal VLAN 42 is mapped to VLAN 144 for its gateway learned from interface Eth 1/3. Please note, the system internal VLAN ID is “local” significant, meaning VLAN ID or VLAN ID range assigned to carrier each data VLAN is different from each individual leaf node.

In summation the output of the command is:

1. Mapping Internal VLAN ID and Data VLAN ID (for leaf1 switch) – VLAN 15 for VMM Domain data VLAN 1044, and VLAN

38 on leaf 2 switch. 2. For EPG “HPE:HPE-ANP:App” in ACI will have two related VLANs mappings

One for VMM Domain - VLAN 15 (on leaf 1) and VLAN 38 (on leaf2) – VLAN 1044 mapping (VLAN 1044 is dynamically assigned when EPG App associates with VMM domain)

One for Physical Domain – VLAN 42 – VLAN 144 mapping (VLAN 144 is statically assigned when EPG App is associated with physical domain and statically bind to interface e1/3 of leaf node 1)

3. Endpoint MAC address, IP address present and learned for VMM Domain interface Port-Channel 8.

2nd useful command is “show vlan extended” “Show vlan extend” command displays internal VLAN ID, EPG, and active VLANs are encapsulated. Leaf1# show vlan extended

VLAN Name Status Ports

---- -------------------------------- --------- -------------------------------

13 infra:default active Eth1/47, Eth1/48

15 HPE:HPE-ANP:App active Eth1/5, Eth1/6, Po7, Po8

17 HPE:HPE-ANP:Web active Eth1/5, Eth1/6, Po7, Po8

18 HPE:HPE-ANP:DB active Eth1/5, Eth1/6, Po7, Po8

23 HPE:HPE-ANP:vMotion active Eth1/3, Eth1/5, Eth1/6, Po7,Po8

41 HPE:App-BD active Eth1/3, Eth1/5, Eth1/6, Po7,Po8

42 HPE:HPE-ANP:App active Eth1/3

43 HPE:vMotion-BD active Eth1/3, Eth1/5, Eth1/6, Po7,Po8

44 HPE:HPE-ANP:vMotion active Eth1/3

45 HPE:DB-BD active Eth1/3, Eth1/5, Eth1/6, Po7,Po8

46 HPE:HPE-ANP:DB active Eth1/3

47 HPE:Web-BD active Eth1/3, Eth1/5, Eth1/6, Po7,Po8

48 HPE:HPE-ANP:Web active Eth1/3

49 HPE:Ext-mgmt-BD active Eth1/3, Eth1/5, Eth1/6, Po7,Po8

50 HPE:HPE-ANP:Ext-mgmt active Eth1/3, Eth1/5, Eth1/6, Po7,Po8

VLAN Type Vlan-mode Encap

---- ----- ---------- -------------------------------

13 enet CE vxlan-16777209, vlan-4000

15 enet CE vlan-1044





23 enet CE vlan-141

27 enet CE vxlan-15302580





42 enet CE vlan-144



46 enet CE vlan-145


48 enet CE vlan-143


50 enet CE vlan-140

Leaf2# show vlan extended

VLAN Name Status Ports

---- -------------------------------- --------- -------------------------------

13 infra:default active Eth1/46, Eth1/47, Eth1/48

35 HPE:Web-BD active Eth1/5, Eth1/6, Po7, Po8

36 HPE:HPE-ANP:Web active Eth1/5, Eth1/6, Po7, Po8

37 HPE:App-BD active Eth1/5, Eth1/6, Po7, Po8

38 HPE:HPE-ANP:App active Eth1/5, Eth1/6, Po7, Po8

39 HPE:DB-BD active Eth1/5, Eth1/6, Po7, Po8

40 HPE:HPE-ANP:DB active Eth1/5, Eth1/6, Po7, Po8

46 HPE:vMotion-BD active Eth1/5, Eth1/6, Po7, Po8

48 HPE:HPE-ANP:Ext-mgmt active Eth1/5, Eth1/6, Po7, Po8

56 HPE:HPE-ANP:vMotion active Eth1/5, Eth1/6, Po7, Po8

VLAN Type Vlan-mode Encap

---- ----- ---------- -------------------------------

13 enet CE vxlan-16777209, vlan-4000

14 enet CE vlan-146



18 enet CE vlan-140


33 enet CE vlan-147

34 enet CE vlan-148










48 enet CE vlan-140


56 enet CE vlan-141

show endpoint detail HPE:HPE-

ANP:App

Use “show interface e1/5 switchport” to confirm active system internal VLANs on the physical domain or VMM domain, “Show interface e1/5 switchport” displays allowed internal VLANs on an interface Leaf1# show int e1/5 switchport

Name: Ethernet1/5

Switchport: Enabled

Switchport Monitor: not-a-span-dest

Operational Mode: trunk

Access Mode Vlan: 21 (default)

Trunking Native Mode VLAN: unknown (default)

Trunking VLANs Allowed: 15,17-18,22-23,41,43,45,47,49 (EPG App is encapsulated in vlan 15)

FabricPath Topology List Allowed: 0

Administrative private-vlan primary host-association: none

Administrative private-vlan secondary host-association: none

Administrative private-vlan primary mapping: none

Administrative private-vlan secondary mapping: none

Administrative private-vlan trunk native VLAN: none

Administrative private-vlan trunk encapsulation: dot1q

Administrative private-vlan trunk normal VLANs: none

Administrative private-vlan trunk private VLANs: none

Operational private-vlan: none

Leaf1# show int e1/3 switchport

Name: Ethernet1/3

Switchport: Enabled





Trunking VLANs Allowed: 41-50 (VLAN 144 is encapsulated in vlan 42)











Leaf2# show int e1/5 switchport

Name: Ethernet1/5

Switchport: Enabled





Trunking VLANs Allowed: 17,35-40,46,48,56(EPG App is encapsulated in vlan 38)











From VM/Host:

From ACI – VMM:

From Agg L3 (Gateway device for network):

DIA-N6K-01# show ip Arp | in 10.16.144.141

10.16.144.141 00:00:10 0050.5689.21ed Vlan144 Gateway device outside ACI learned MAC in VLAN 144.

DIA-N6K-01# show mac address-table | in 21ed

* 144 0050.5689.21ed dynamic 20 Eth1/3 MAC learned from interface e1/3

From a device (remote / routed to Data Center LAN) – VM App1 reachable from anywhere on the LAN.

C:\Users\Administrator>ping 10.16.144.141






Ping statistics for 10.16.144.141:

Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),

Approximate round trip times in milli-seconds:

Minimum = 37ms, Maximum = 381ms, Average = 150ms

Another useful command is “show vPC” to see if port-channel membership and VLAN allowed. It will be covered in next section. The below is a table with detailed VLAN mapping derived from summarizing the outputs of the prior commands.

L12-P5-vPC-PG

Leaf1# show lldp neighbors

Device ID Local Intf Hold-time Capability Port ID

VC2040F8-7C9519001J Eth1/5 120 X5

VC2040F8-7C9519001K Eth1/6 120 X5

Leaf2# show lld neighbors

Device ID Local Intf Hold-time Capability Port ID

VC2040F8-7C9519001J Eth1/5 120 X6

VC2040F8-7C9519001K Eth1/6 120 X6

From Fig 38, Interconnect section of OneView’s display shows VC1 has serial number 7C9519001J while VC2’ serial number is 7C9519001K.From the output of the “show LLDP neighbor” leaf node CLI command, interface eth1/5 on both leaf1 & leaf2 connects to VC1: eth1/5 on leaf node 1 is connected to port X5 on VC1 and eth 1/5 of leaf node 2 is connected to port X6 of VC1.The Same method is used to connect interface eth1/6 on both leaf1 & leaf2 to VC2 - eth1/6 on leaf swich2 to port X5 on VC2 and eth 1/6 of leaf node 2 is connected to port X6 of VC2.

VC 1 terminated on the different leaf node but landed on the same interface e1/5 on the leaf node 1 & 2.

To confirm vPC is up running and all the data VLANs are allowed on vPC links. Leaf1#show vpc

Legend:

(*) - local vPC is down, forwarding via vPC peer-link

vPC domain id : 120

Peer status : peer adjacency formed ok

vPC keep-alive status : Disabled

Configuration consistency status : success

Per-vlan consistency status : success

Type-2 consistency status : success

vPC role : secondary

Number of vPCs configured : 4

Peer Gateway : Disabled

Dual-active excluded VLANs : -

Graceful Consistency Check : Enabled

Auto-recovery status : Enabled (timeout = 240 seconds)

Operational Layer3 Peer : Disabled

vPC Peer-link status

---------------------------------------------------------------------

id Port Status Active vlans

-- ---- ------ --------------------------------------------------

1 up -

vPC status

----------------------------------------------------------------------

id Port Status Consistency Reason Active vlans

-- ---- ------ ----------- ------ ------------

346 Po2 up success success 4000

690 Po1 up success success 4000

691 Po7 up success success 140-141,1042,1045,1048-1050

692 Po8 up success success 140-141,1042,1045,1048-1050 Verify vPC 691 connections from APIC GUI as a demonstration.

vPC Interface 691 & 692 – leaf 1 vPC Interface 691 & 692 – leaf 2

Verify and display the details for connection eth1/5 on leaf1 and leaf2 switches.

vPC Interfaces (vPC 691) allows physical domain and VMM domain VLANs on the trunked interfaces – same for Leaf 1 & 2

From two set of screenshots shows, vPC port channel #691 are used for eth1/5 on both leafs switches and used same policy group. (L12-P5-vPC-PG). These vPC interfaces (eth1/5 on leaf 1 & 2) are trunked (switched mode trunk) and allows the same set of VLANs (VLAN 140,1041,1044-1045,1050) to pass through.

Aggregated Port-Channel (PO 7) allows leaf internal VLANs on trunked interfaces – Different on leaf 1 & leaf 2.

Aggregated port-channel 7 interface is running “outside” of vPC 691 and is assigned a number of VLANs to “carry” (encapsulate) these data VLANs through the Fabric. As we discussed earlier, these VLANs are assigned by the leaf node and they are internal/locally significant. As these output are shown above, leaf node 1 uses VLAN range 15,17,18,21,41,43,45,47,49,50 versus VLAN range 15-

16,18,21,36-37,40-43,46-47 on leaf node 2.

Best Practice of vPC connections placement on leaf nodes

By connecting uplinks port X5 and X6 of VC1 at the same port on both leaf nodes (eth 1/5), only one switch profile is needed and one “L12-P5-vPC-PG” policy group can governor behavior of both ports. It simplifies ACI configuration and eases the troubleshooting.

From the LACP Interface of eth1/5 interface configuration screen (at the bottom portion of the display) we can confirm both eth 1/5 interfaces are terminated on the same VC1 with system ID “7C9519001J”. To summarize, the following diagram showed the detail of port mapping, VLANs active on the different interfaces and policy group assigned to the interface.

LLDP Tagging verificationThis section is to confirm that VLAN IDs are propagated from the ACI infrastructure down to the host and the DVS provides two ways communication.

To verify LLDP tagging configurations and that LLDP communication between vSphere host and ACI are end to end:1. OneView – LLDP tagging feature must be supported and option enabled/Selected

2. ACI – vSwitch Policy in Attachable Entity Policy view – make sure LLDP enabled towards to DVS.

To verify LLDP policy click , a pop-up screen comes up and displays named “LLDP-Enabled-Bidirectional” policy details.

3. vSphere vCenter Server – DVS (advanced – Edit Settings) - LLDP enabled in Discovery protocol section

4. ACI to vSphere host end to end LLDP visibility confirmation – at vSphere vCenter, from network, host uplink “vmnic0” or

“vmnic1” connected to the DVS, click information icon to expand detail information window, from “”All” tab or “LLDP”, At APIC, from VM Network, expand…..

Click the information icon next to vmnic0 to expand connection information and Click LLDP tab to see the details.

From the screenshot above verified between vSphere virtual distribute switch passing LLDP information and connecting to VC1.Leaf 1# show lldp neighbors interface e1/5 detail

Chassis id: 50:65:f3:5e:a5:79

Port id: X5

Local Port id: Eth1/5

Port Description: HP VC FlexFabric-20/40 F8 Module 4.45 X5

System Name: VC2040F8-7C9519001J

System Description: HP VC FlexFabric-20/40 F8 Module 4.45 2015-07-21T00:33:55Z

Time remaining: 88 seconds

System Capabilities:

Enabled Capabilities:

Management Address: 10.16.40.103

Vlan ID: not advertised

Total entries displayed: 1

The vmnic1 is connecting to VC2 as we can confirm by its system name.

Leaf node 1# show lldp neighbors interface e1/6 detail

Chassis id: 50:65:f3:5e:a5:85

Port id: X5

Local Port id: Eth1/6

Port Description: HP VC FlexFabric-20/40 F8 Module 4.45 X5

System Name: VC2040F8-7C9519001K

System Description: HP VC FlexFabric-20/40 F8 Module 4.45 2015-07-21T00:33:55Z

Time remaining: 100 seconds

System Capabilities:

Enabled Capabilities:

Management Address: 10.16.40.103

Vlan ID: not advertised

Total entries displayed: 1

VSphere host learned VLAN ID 1044 (for example) from VCs through LLDP tagging feature, VMs will see EPG associated portgroups

available as a choice of their network choices. This concludes downstream (between VDS and VC) LLDP communication verification.

Solution Resilience TestsThe purpose of the tests is to validate design principles and results of interruption to each fault domain. Three fault domains will be tested as VC1, VC2, or leaf node 2.

Testbed:

A testing VM-1 is App1 on an ESXi host integrated into an ACI VMM infrastructure. IP address of the VM is at 10.10.144.41/24. The second VM “testing VM-2” is on an ESXi host outside ACI at 10.16.43.243. The network for “testing VM-1” is assigned to portgroup “HPE|HPE-ANP|App” on an ACI created VDS. The network of “Testing VM-2” is assigned to data center management VLAN. Inter-VLAN routing is through aggregate switches. A ping utility was used with the following config:

Trace interval – 1 ping per second

Statistics Samples to include – 1

Packet Size (in byte) – 1,000.000

Ping continuous from both directions during test, (From Testing VM-1 VM2 and Testing VM-2 VM1)

Before failover tests were started, the connectivity was verified. Ping from testing VMs was successful. The following scenarios were tested:

Simulating VC2 Down by powering off VC2Action: Shutdown VC2 module via Virtual Power Button in OAResult: 1 -2 ping loss was observed. See Appendix B for detailed data and graph.

Simulating a VC1 Down - by powering off VC1Action: Shutdown VC1 module via Virtual Power Button in OAResult: No immediate ping loss, but after the module was down 2 -3 minutes, a delayed ping loss occurred. Total ping lost 3 – 5. Please see Appendix B for detailed data and graph.

Result: No ping loss was observed when leaf node 2 was powered down, but some ping loss occurred when the fabric re-converged.

See Appendix B for detailed data and graph.

h20628. · 2019-01-17 · these vlans are configured on this l3 switch as svi for its network ip...

Documents