DELIVERING OPENSHIFT ON OPENSTACK FOR PERFORMANCE AND SCALESTEVE GORDON (@xsgordon)

GOAL

● 1000 OpenShift Container Platform nodes; hosted on● 300 physical Red Hat OpenStack Platform nodes

2

3

APPLICATION CONTAINERS

1. Code: mysqld

2. Configuration: /etc/my.cnf

3. Data: /var/lib/mysql

4. Other stuff :-)

CODE, CONFIGURATION, DATA

4

EXPOSITION OF RESOURCES

5

CONSUMPTION OF RESOURCES

OPENSHIFT ON OPENSTACK

6

Architectural tenets:

● Technical independence● Contextual awareness● Avoiding redundancy● Simplified management

Reference architecture:● red.ht/2ibNmvX

PERFORMANCE AND SCALE TESTING

7

WHERE TO TEST?

8

The CNCF cluster is made up of 1000 nodes deployed at Switch, Las Vegas by Intel for the use of the CNCF community:

Compute Node Spec● 2x Intel E5-2680v3 12-core● 256GB RAM● 2x Intel S3610 400GB SSD● 1x Intel P3700 800GB NVMe PCIe

SSD● 1x QP Intel X710

Storage Node Spec● 2x Intel E5-2680v3 12-core● 128GB RAM● 2x Intel S3610 400GB SSD● 10x Intel 2TB NLSAS HDD● 1x QP Intel X710"

Got a nefarious plan for taking over the world using CNCF related open source projects? Head to https://github.com/cncf/cluster.

WHAT TO TEST?

Goal: 1000 OpenShift Container Platform nodes, on 300 Red Hat OpenStack Platform nodes● Push deployment to its limit, identify:

○ Bottlenecks,○ Config changes, and○ Best practices

● Document, file, and fix issues as appropriate.

9

ARCHITECTURE

10

● OpenStack UnderCloud

ARCHITECTURE

11

● OpenStack UnderCloud● OpenStack Overcloud control

plane

ARCHITECTURE

12

● OpenStack UnderCloud● OpenStack Overcloud control

plane● OpenStack Overcloud compute

ARCHITECTURE

13

● OpenStack UnderCloud● OpenStack Overcloud control

plane● OpenStack Overcloud compute● OpenShift/Kubernetes master node

VMs● OpenShift routing, registry, and

metrics● OpenShift/Kubernetes minions● Load Generators

ARCHITECTURE

14

● OpenStack UnderCloud● OpenStack Overcloud control

plane● OpenStack Overcloud compute● OpenShift/Kubernetes master node

VMs● OpenShift routing, registry, and

metrics● OpenShift/Kubernetes minions

ARCHITECTURE

15

● OpenStack UnderCloud● OpenStack Overcloud control

plane● OpenStack Overcloud compute● OpenShift/Kubernetes master node

VMs● OpenShift routing, registry, and

metrics● OpenShift/Kubernetes minions● Load generators

SYSTEM VERIFICATION TEST SUITE

● Red Hat OpenShift Performance and Scalability team’s upstream test suites● Main tests are

○ cluster-loader○ Networking/synthetic○ Workload Generator○ Reliability/Longevity

● https://github.com/openshift/svt

16

17

CLUSTER-LOADER ARCHITECTURE

Start Parse args & config

ConfigObj

End

False

Create Namespace

True

XExists

?Items < N

False

Create X

Iterate ItemCount

True True

False

X can be:● Quota● Template● Service● User● Pod● RC

CLUSTER LOADER TEST

18

OpenShift (Kubernetes) Object Quantity

Nodes 1,000

Namespaces (projects) 13,000

Pods 52,000

Build Configs 39,000

Templates 78,000

Image Streams 13,000

Deployment Configs and Services 39,000 (incl. 13,000 Replication Controllers)

Secrets 260,000

Routes 39,000

Component # of bugs filed/encountered

Kubernetes 8

Installer 8

Images 5

Docker 5

Containers 3

Storage 2

Ansible 2

Management 1

iptables 1

Deployments 1

Command 1

Builds 1

Auth 1

NOTABLE SCALE ISSUES ENCOUNTERED

19

etcd Disk Utilization API Server CPU

NOTABLE SCALE ISSUES ENCOUNTERED

20

Ansible Installer https://github.com/ansible/ansible/issues/16749

FUTURE OPPORTUNITIES

● Reference architecture available at red.ht/2ibNmvX● Containers and OpenStack: A platform for distributed applications, paper

available at http://red.ht/2hSfIPs● Learn more about Red Hat Summit at redhat.com/summit

21

THANK YOU

22