Jumbo Mumbo in OpenStackVikram HosakoteCisco Systemsvhosakot@cisco.com

OpenStack Summit

Barcelona

Agenda

• What is MTU?• IP Fragmentation• Jumbo Frames in OpenStack• Use Case• Path MTU Discovery• Advantages & Disadvantages of Jumbo Frames• Q&A

3

What is MTU?

4

• MTU (Maximum Transmission Unit) is the size (in bytes) of the largest packet that can be transferred by an interface without IP fragmentation

• MTU is• Fixed before transmitting data (Ethernet)• Negotiated during handshake/connect time (point-to-point serial links)• Dynamically determined on-the-fly while transmitting data

• Default MTU is 1500 bytes

• Jumbo MTU is 9000 bytes

• What is MTU?

5

• What is MTU?

6

IP Fragmentation

7

• “IP fragmentation is an process that breaks datagrams into smaller pieces (fragments), so that packets may be formed that can pass through a link with a smaller maximum transmission unit (MTU) than the original datagram size. The fragments are reassembled by the receiving host.”

• https://en.wikipedia.org/wiki/IP_fragmentation

• Adds additional processing overhead on the interface

• IP Fragmentation

8

• IP Fragmentation (DF bit in IP header)

9

Jumbo Frames in OpenStack

10

In neutron.conf[DEFAULT]

global_physnet_mtu = 9000

advertise_mtu = true

In openvswitch_agent.ini [ovs]

bridge_mappings = provider1:eth1,provider2:eth2,provider3:eth3

In ml2_conf.ini[ml2]

physical_network_mtus = provider2:4000,provider3:1500

path_mtu = 9000

• Jumbo Frames in OpenStack

11

• Enable the DHCP MTU option (26) in /etc/neutron/dnsmasq-neutron.conf.• dhcp-option-force=26,1454

• Restart dnsmasq

• MTU values only apply to new network resources

• network_device_mtu in nova.conf is deprecated in Juno

• Jumbo Frames in OpenStack

12

Use Case

13

• Boot a nova instance

• Create a cinder volume using ceph backend

• Attach volume to instance and mount it

• SSH into instance and write lots of data in this volume

• Measure traffic drop on ceph nodes

• Use Case

14

15

16

[mc-compute-1 ~]# ip a | grep mtu2: mx: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 90003: t: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 90004: p: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 90007: br-inst: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UNKNOWN8: br-prov: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UNKNOWN9: br-int: <BROADCAST,MULTICAST> mtu 9000 qdisc noop state DOWN10: phy-br-inst@int-br-inst: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 900011: int-br-inst@phy-br-inst: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 900012: phy-br-prov@int-br-prov: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 900013: int-br-prov@phy-br-prov: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000

MC-UCS-FI-A(nxos)# show queuing interface | grep MTU q-size: 360640, HW MTU: 9216 (9216 configured)

17

[mc-block-1 ~]# ping -M do -s 8972 10.7.8.2PING 10.7.8.2 (10.7.8.2) 8972(9000) bytes of data.8980 bytes from 10.7.8.2: icmp_seq=1 ttl=64 time=0.118 ms8980 bytes from 10.7.8.2: icmp_seq=2 ttl=64 time=0.066 ms--- 10.7.8.2 ping statistics ---

2 packets transmitted, 2 received, 0% packet loss, time 3999ms

rtt min/avg/max/mdev = 0.062/0.082/0.118/0.022 ms

-M used to select Path MTU Discovery strategy. May be either do (prohibit fragmentation, even local one), want (do PMTU discovery, fragment locally when packet size is large), or dont (do not set DF flag)

18

Path MTU Discovery

19

• Path MTU Discovery (PMTUD) is a technique to determining the MTU of the network path between source and destination.

• PMTUD works by setting the Don't Fragment (DF) flag bit in the IP headers of outgoing packets. Then, any device along the path whose MTU is smaller than the packet will drop it, and send back an Internet Control Message Protocol (ICMP) Fragmentation Needed (Type 3, Code 4) message containing its MTU, allowing the source host to reduce its Path MTU appropriately.

• The process is repeated until the MTU is small enough to traverse the entire path without fragmentation.

• Path MTU Discovery

20

Advantages & Disadvantages

21

• Advantages• Greater efficiency• Lesser packet drop• Interfaces process fewer packets

• Disadvantages• Affects per-packet processing• Bigger packets on the wire may increase lag and latency• Packet corruption causes huge retransmits and congestion

22

Q & A