cloud foundry summit 2017

Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .

2017年6月21日

The Road to "JYU-BAI"- Adopting Cloud Foundry at Yahoo! JAPAN -

2017年6月20日


About me

2

Software Engineer

Manager

Yahoo! JAPAN

Yasuhiko Kubono

Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .3

• Introducing Cloud Foundry into our

services

-Yasuhiko Kubono

• How do we Actually Operate

-Yusuke Kondo

Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .Copyrig ht © 2017 Yahoo Japan Corporation. All Rig hts Reserved .

Introducing Cloud Foundry

into our services


Agenda

5

• About Yahoo! JAPAN

• Why we use Cloud Foundry?

• Introducing Cloud Foundry into our services

• Case study


About

Yahoo! JAPAN


Outline

Yahoo Japan Corporation(SoftBank Group)

Businesses: Internet Advertising - e-Commerce

- Member services - Others

Headquarters: Tokyo Japan

Founded: January 31, 1996

# of Employees: 5,826 (As of March 31, 2017)

7


# of Engineers & Designers

2,500


More than 100

Web Services


39,89M Active User IDs

1Month※2017年1-3月の平均

67,4B Page Views

Total requests


Why we use

Cloud Foundry?


Why we use Cloud Foundry？

Speed up development time

JYU-BAI increase productivity by 10 times

12


Adoption Plan

13

2016

Initial introduction to

a few services

2017

Full-scale

implementation

2018

Expand

implementation

Here


Introducing

Cloud Foundry

into our Services


More than 100

Web Services


Programming Languages

16

C, Perl, C++,

PHP, Node.js, Java...


Architecture Differs by Web Service

17

e.g. Travel tips e.g. Yahoo! Auction

Small-scale web services

list

cart

logic

logic

logic

Large-scale web services

Search

API Gateway


Obstacles

18

Same architecture does not fit

in each web services


Solutions

Enroll CF Coach in each web servicesaround 20 staff / 15 services

Coaches role:Promote cloud design methods that suit for each web services

19


Role map

20

Core Team

CF Coach

For

Shopping

Shopping

engineer

Shopping

engineer

CF Coach

For

Auction

Auction

engineer

CF Coach

For

Media

Media engineer

…


Case study


Where we started from


List Necessary Functions

23

Service A Service B Service C Service D Service E Service F Service G

MySQL ● ● ● ●

Oracle ● ● ●

KVS

ObjectStrage ●

C/C++ ●

PHP ● ● ● ●

Node.js ● ● ● ● ●

Java ● ● ●

advertisement ● ● ● ● ●

beacon ● ●


Challenges we encountered

Functions that can’t be used in the cloud because of

complicated dependency

Internal security polices are not suited for cloud

environment

Most of our web services were stateful design24


How we started

We selected one web service, and started

by preparing the necessary functions for

that service

Resolve issues each time they occur

25


So, which web servicedid we start with?


Criteria for the web service

1. Simplicity

• Service with limited functions and external PF that can be used

2. Actively developed

• Web services that actively developedso that the effectiveness of introducing CFcan be measured

27


First target : CS tool

• Constructed with few servers in OpenStack environment

• WebServer: apache

• Apache Traffic Server (ATS) : Reverse Proxy

28

Characteristics• Language: PHP

• Framework： cakephp

• Uses REST API

• MySQL

Server Configuration

ATS

CS tool(apache)

API

ATS

MySQL

HTTPS

HTTP MySQL

CS tool(apache)

(our auction service)


Partial release using ATS (entry points)

Partially diverted entry point to CF apps using ATS:

• So that CF or OpenStack can be switched in entry points

29

ATS

CS tool(apache)

CS tool(apache)

API

HTTPS

HTTP

Some entry points

ATS

CS tool(CF)


Lessons learned from the first target

• How to Implement in Production

• Development method based on OSS

• How to make service stateless on CF

30


Adopting & Expanding to other services

31

Decide

target

Investigate

issues

Adopting

knowledge Solve issues

Knowhow

accumulated


Next Presentation,

How do we Actually Operate

Photo by: Aflo


Hello CF Summit 2017!

Yusuke Kondo or @konfoo

Responsible to...• operating Cloud Foundry & Concourse on IaaS• increasing engineers’ productivity by providing

tools and best practices around CI/CD


Overview of Yahoo! JAPAN proprietary Infrastructure

More than four DCs in Japan

More than 90,000 VM running on OpenStack

34


Cluster Spec

35

dev production

Load Balancer Software Hardware x2

IaaS Openstack Openstack

Hypervisor # 40 40


Current Status (As of Jun. 9, 2017)

36

dev production

Cluster # 1 1

Cell # per Cluster 40 30

Org # 136 38

App Instance # approx. 2,000 approx. 400

Rps at peak time N/A approx. 2,000


Future Plan (As of Jun. 9, 2017)

37

dev production

Cluster # 1 1 => 6

Cell # per Cluster 40 30 => 100

Org # 136 38

App Instance # approx. 2,000 approx. 400

Rps at peak time N/A approx. 2,000


Integration

with

Backend Services


App Role Based ACL

MQ

Service

RDB

Object Storage

Key Value Store

Cache Service

FaaS

Existent Platforms

39


Integration with Existent Platforms

• Cookie off-loading Route Service

• On-demand MySQL (OpenStack Trove API)

• Distributed pubsub service (Pulsar)

40


Marketplace Dashboard

41

Goal: Providing all PFs in CF Marketplace


Issues we faced

Platform ACL is based on IP address or hostname

=> Requesting for exceptional permission for accessing via IP Range with a limited term.

=> Migrating from Host-based ACL to Role-Based ACL in the long term

42


Integration

with

Logs and Metrics


What we already have

In-house Monitoring & Alerting PF based on Apache Kafka, Hbase,

Splunk, an enterprise log analytics platform

44


User-side Logs and Metrics

45

PCF Cluster-1

VM

APP

VM

App

VM

VM

APP

App

Loggregator

Splunk

Monitoring PF

App AppApp

APPAPP

APPAPP

No action is needed for App developers


What we prepared

Firehose Nozzle and Relay Server

• Nozzle filters and formats the App logs streamed by Firehose

• Relay Server forwards the log stream to specific index

46


Issue we faced

High log traffic. 900 lines per sec ! (as of Jun. 8, 2017)

=> Provided users with CF friendly logger

47


Operator-side logs and Metrics

Splunk

• Platform logs such as CF component syslog

Prometheus

• Bosh metrics, VM metrics, Firehose metrics

• Emitting alerts to our smartphone

48


Integration

with

other System


Integration with package monitoring tool

51

Application Source Code

Dependent packages

Runtime Buildpack version

Vulnerable Package

Monitoring Tool

Track the buildpack version which the App are staged with and report outdated apps.


Integration with package monitoring tool

52

Application Source Code

Dependent packages

Runtime

Scan package version

Scan whole source code


Integration with Concourse

We use Concourse for

• deploying new Cloud Foundry release

• updating buildpacks

• syncing employee accounts with UAA

• backup database to object storage

• ...

53


Lessons learned


We are still on the way to change mind

Changing your organization mind is the most essential part.

• Educate not only users, but also platform division where you belong.

• Work closely with your security paranoid team. Involve them to update the policy

55

cloud foundry summit 2017

Technology