Concur, an SAP Company

De-Centralizing Operations with APMKevin Evans, VP of DevOps and Cloud Services, Concur

Angus Claus, Director Service Management, ConcurDonnell Baker, Sr. Manager SRE, Concur

Concur, an SAP Company

De-Centralizing Operations with APM

Kevin Evans, VP of DevOps and Cloud Services Angus Claus, Director Service Management and SRE Donnell Baker, Sr. Manager SRE

November 2016

DevOps @ Concur

A very brief review of the Concur service delivery journey

SDLC* methodology

Release cadence

Architecture

Ops model

* Systems development lifecycle

2003

Waterfall

9 – 18 months

On premise as a service

IT ops

2008

Mixedwaterfall / agile

Initial: 4 monthsEventual: 1 month

Monolithic hybrid

Centralizedhosted ops

2014

Agile

Initial: 1 monthEventual: 1 day

Microservice aspirations

Evolving End 2 EndDevOps

Foundational principles

• SaaS == Software as a Service – We are building a service, not just software

• Decentralization and empowerment– Decompose system into a set of services – Dedicated team owns each service – Team is enabled and responsible for every aspect of that service

• Simplicity, simplicity, simplicity

Motivation for transitioning to the DevOps model

Why?

Innovationvelocity

Operationalaccountability

Architecture change

Core Practices• Automation is Key to everything – Pets vs. Cattle

– Scale, Consistency, Velocity, Manageability

• Engineering vs. Administration

– Centralized group focus moves from managing servers and infrastructure to building tools that enable others to do this work

• Frictionless and the lesson of the free Market

– Mandates don’t exist. Make the desired path the easiest to achieve compliance

• Build Culture where Everyone is accountable to the Service

– Direct ownership enables teams to be accountable for their service

• Learn from Others

– We are not the first ones to travel this path, what have other companies done to solve the problems. How can we take these learning's, adjust, and apply

What does DevOps mean at Concur?

End-to-end (E2E) teamsHigh degree of freedom

• Technology• Velocity of release

Self-service for operational services

Operationally accountable for service

• Performance

• Availability

• Security

• Quality

• Cost of Ownership

Customers Auditors Go to Market

E2E Teams

E2E Teams

E2E Teams

E2E teams

E2E Teams

E2E Teams

E2E Teams

E2E teams

E2E Teams

E2E Teams

E2E Teams

E2E teams

Limitedcentralizedownership

Delivery pipeline

SecurityQE CloudServices

Production environments

Enabling DevOps with New Relic

10© 2015 SAP SE or an SAP affiliate company. All rights reserved. 10© 2015 SAP SE or an SAP affiliate company. All rights reserved. 10© 2015 SAP SE or an SAP affiliate company. All rights reserved.

How We Champion APM

Empowered Teams

Expert Services

The“Rails”

CustomerSuccess

Why Do Rails Matter?

Our Strategy - The “Rails”

Centralized Control and Standardization

• Standard Naming Convention– App Names– Labels

• Developers are First Class Citizens– API Driven Configuration

• Centralized services– Plug-in Abstraction

Our coverage

UI

MT

DB

4 Billion Transactions/Month

460+ Unique logins from 96 Agile teams

New Relic charts and data used in weekly Service Reviews

99%+ of Major incidents reference New Relic data

8,700+ of Alerts/Month40,000+

519 Servers

80 Apps

2 Plug-ins

436 Servers

29 Apps

1 Plug-ins

4 Plug-ins

The Future of APM @ Concur

Minimum Bar

• Automated operational maturity

Zero Touch Configuration

• Provide - Configuration as Code:– Role Type– SLA KPIs (Apdex Thresholds)– Escalation Path (Pagerduty)

• You get Out of Box:– Default Alarm / Notification Channel Set Up– Basic Alarming– Monitored SLA (Apdex)– Dashboards

• Custom: – Bespoke Alarms – Error Rates, Response Times, Min / Max Throughput– Error Code Exclusion– Auto Remediation

What Will Keep Us Successful?

• Role Based Access Control (RBAC)• Pipeline Delivery (Control Plane)

– Build– Ship– Run

Q&A