scaling agile in ericsson oss -...

Slide subtitle

John Coogan, Oct 2016

Scaling Agile in Ericsson OSS

Ericsson Internal | 2016-09-20 | Complex Networks, Simple Management

Ericsson oss

›Operator Support System

›Real Time

›1 Billion Subscribers

›320 Operators

›Over 1000 Deployments

Ericsson Internal | 2016-09-20 |

OSS Made Easy

F

C

A

P

S


›OSS-RC to ENM

›A New Strategy for a New

Product

Ericsson Athlone


ENM Development

5 countries

100+ Teams

~1400 Engineers

1880M2

Data Centre


Scaling Agile


Organisational Challenges

› Strong focus on individual teams

› Parallel development of

framework, platform and SW

› Communication: Vision gets lost

› Main requirement transparency

› User story size and team

capability to deliver


ENM Program Structure

Administration

Program Management

Stakeholder Management

Requirement Area 1

Early Phases

Forecast &

Capability

Requirement Area 4 Requirement Area 5

Bug Follow Up

Improvements Program Communications

Operational PO

Flow Vizualisation

& Optimisation

Technical

Coordination

Integration

PlanningRelease

Validation

Maintrack

ManagementProduct

Introduction

Requirement Area 3

Quality

ManagementEngineering

Tools

Requirement Area 6Requirement Area 2

Area POs

Test

Manager

• MT Stand-up

• Outlook

• Bug Akuten

• Program

Steering


Ready - Set - Go

Ready (15B)

2015-09-24

Customer Lab

5K cells (1250 Nodes) Support

LTE eRBS (G1)

• FCAPS

• AMOS

Set (16A)

2015-12-15 (PD4)

Live network (LA)

15K cells (3750 Nodes)Support

LTE eRBS (G1)

• FCAPS

• Auto Provisioning

• FMX

Go (16B)

2016-05-06 (PD4)

GA Product

40K cells (10,500 Nodes) Support

LTE eRBS (G1+G2)

• Macro/Micro/RDS/PICO

SGSN-MME/vSGSN-MME

IP Router 6000

MINI-LINK


Poster Campaigns

ENM Info Session

TV

communication

Telepresence


ENM Main Requirement Wall

Slide subtitle

3 week sprint calendar

Analysis

Deliver to MT

Coding

5 dayavg US

cycle time

Code review

Await code review

Transaction Cost (→ 0)

Wait time (→ 0)

Config TE & run test

5 dayavg US

cycle time

5 dayavg US

cycle time

Fast Commit


› Automated test environment

spin up and test suites: From

days to <1 hr

› Technology– Docker

– J2EE UT in secs

– Arquillian

– TAF

› Coaching for 5 day User

Stories

Fast Commit Enablers

Slide subtitle

Test in a scaled agile environment

Ericsson Internal | 2016-09-20 | Page 16

Agile Quadrants

Q1

Q2 Q3

Q4

Business Facing

Technology Facing

Su

pp

ort

ing th

e T

ea

mC

ritiqu

e th

e P

rod

uct

Static Tests

Unit Tests

Basic Integration Tests

Acceptance Tests

Usability Tests

Exploratory Tests

Security Tests

Performance Tests

Load Tests

ility Tests

Automated

TAF

Automated

Dev Env

Manual and

Automated

Tools

TAF


›Whole Team Approach

›Team Maturity

›Test Community

›Support the Teams to

Produce Quality

Test Role


ENM Development flow

IDE

IDE

IDE

BIT

Tests

BIT

Tests

BIT

Tests

Maintrack

Entry

RFA

(Long Loop)

Instant <10 mins

Integration

<3 hour

Main Track Entry (MTE)

Tests

<8 hrs

All teams Tests

RFA Level of Test

RVB

72 hrs

RVB Level of Test

Visualisation and Reporting

Unit

Tests

Unit

Tests

Unit

Tests

seconds

Unit

TAF

Test

TAF

Test

TAF

Test

<10 mins

Functional Test

Main trackTEAM Environment

.

.

.

.

.

.

.

.

.

.

.

.


Enm Maintrack Loops

Short Loop Long Loop

<3 Hr <8 Hrs

Load Test

Main track LOOPS

Sprintupgrade

Upgrade

Under load

functional

level test

5k cells,

degrade

detection

All teams Test

<24 Hrs

Cloud

Cloud

Deployment

Robustness

Customer

Upgrade

LOOP

<72 Hrs

60k Cells,

KPIS

Availability

BUR

Backup

& Restore

<1 Wk

Sprint

Release

Real NODE

Manual

E2E

Workflow

• Over 1000

Commits/

sprint

• 25 physical

ENM envs

• Daily

standup

Feedback to

Teams

24 / 7

operation

Follow the sun

for blocking

Issues

Continuous

innovation –

Shift Left


MT Loop Evolution

Basic

FCAPS Upgrade

480 Node Upgrade

under load

(5K Network)

RFA250

2015 2016

Qu

ality

Bar

Degrade

Detection

Q4 Q1

Upgrade

under load (15K Network)

Q2 Q3

Availability Fast MT

Q4


Test as a Service

Node Access

Cloud Based Test Environment

Simulated Network

Test Consultancy


Release Validation

automated Load profiles simulate customer conditions. Used in RV but also by the teams

KPI, stability and endurance testing on the latest MT SW. RV Provides a 72 hour feedback to design

Validation of each sprint on a continuous basis using stamping tests and Clean room.

Workflow Test and usability focus

High availability testing and support to ENM teams via automated test Suite

Test Expertise, Supporting the Teams


› Scrum

› Continuous Everything

› Quick to Change

› Strong, Competent Teams with Quality Focus

› Sustainable SW Producing Machine

› Working Software

› Quality First

› Deliver Value to the Customer Frequently

Agile Principles Intact!


› We have been on an adventurous journey

› Scaling agile means re-thinking how you are organised

› Test in a scaled agile environment mean supporting your

team to produce quality

› We have failed many times, but we have failed fast

› We have made it work

› Deliver Value to the Customer Frequently

Conclusion


Questions

scaling agile in ericsson oss -...

Documents