service production from d3 pitfall viewpoint

SERVICE PRODUCTION

Walter Liu

2012/06/11

04/10/2023Confidential | Copyright 2012 Trend

Micro Inc.1

About me• Architect in Core Tech WRS

• Trend Micro 2007~ Now

• Chief Technical Director in Netgame Dep.• Softstar Inc. Taiwan 1998~2007

• Expertise:• Backend service development and operation

It’s coming ……

The beginning

04/10/2023Confidential | Copyright 2012 Trend

Micro Inc.4

Assault - Error 12 !!!

04/10/2023Confidential | Copyright 2012 Trend

Micro Inc.5

The nightmare Error 37

Out of stock in Taiwan D3 packages

• D3 packages are out of stock in largest 3 convenient stores in Taiwan and all game shops.

• Some people shared they finally got packages in Jibei island.

• Many people shared they pursued after trucks of convenient stores to get the packages.

04/10/2023Confidential | Copyright 2012 Trend

Micro Inc.7

Out of Stock of Taiwan Game Card

04/10/2023Confidential | Copyright 2012 Trend

Micro Inc.8

Blizzard Korean lower priority of Taiwan IP

Korean IP

Taiwan IP

Questions?

04/10/2023Confidential | Copyright 2012 Trend

Micro Inc.11

What’s the most important things to users?

04/10/2023Confidential | Copyright 2012 Trend

Micro Inc.12

What mistakes Blizzard make in this D3 service production？

Why Quantity Estimation?

Cost EffectiveHow?

What if it is wrong?

Scalability & Elasticity

Horizontal Infrastructure

Fail in Scalability

0 1 2 3 4 5 6 7 8 9 10 110

200

400

600

800

1000

1200

Performance

EPIC Fail in Scalability

0 1 2 3 4 5 6 7 8 9 100

200

400

600

800

1000

1200

Performance

Ideal Horizontal Scalability

0 1 2 3 4 5 6 7 8 9 10 110

500

1000

1500

2000

2500

Performance

Elasticity - Unpredictable Traffic

Elasticity - Cloud Solution

Elasticity – Others• Elastic Application Architecture.• Several flexible hardware providers.• Flexible ISPs and pricing.• ……

Customer Service and Social Communication

• Bz is doing pretty bad.• FB Event: Closed-beta account in Taiwan.• Build the image to help their users. Like,

• No explanation about incidents.

04/10/2023Confidential | Copyright 2012 Trend

Micro Inc.21

VS.

Incident happens

- Especially your service goes production.

Avengers Assemble !!!

Fast and Responsive Organization

Teams

AwarenessEffective Communication

Fast and Responsive Process• Incident management• Problem management

Some other practices• Interlock with related teams at beginning/middle.

• Customer service prepares resources for burst incoming calls/tickets.

• Customer service prepares training for the new service/product/• Data Center team gives out advices and plan for your project.

• Recruit a Service Manager• Fail case: something that not belong to any team.• Have someone responsible for whole service.

• Update/patch/change SOP• Fail case: Service changed, but your CS don’t know about it. Your

customers are confused when they call your CS.• Fail case: Service changed and caused some trouble, but your

service manager said he didn’t decide/say it.

Not related to these Diablo 3 failures, but important to any system• Availability• Security• Easy to administrate

• System Health/Statistics Monitoring• Easy Deployment• Easy Configure

• Risk Management

04/10/2023Confidential | Copyright 2012 Trend

Micro Inc.27

工商服務時間• Web Reputation Service

• Parental Control and Productivity Control• 像是 Hinet色情守門員，防毒軟體的孩童防護鎖• 或者公司不希望員工上班看色情、賭博等網站

• Web Threat Protection• 防止惡意網頁如病毒 /木馬下載、釣魚網站等等。

• Advanced Persistent Threat• 鎖定特定目標• 假冒信件或者其他• 低調且緩慢• 客製化惡意元件• 安裝遠端控制工具• 傳送情資

04/10/2023Confidential | Copyright 2012 Trend

Micro Inc.28

WRS – Parental Control

WRS – Web Threat Protection

Funny Diablo 3 sales on Taobao, China

04/10/2023Confidential | Copyright 2012 Trend

Micro Inc.31

Thank You!

Risk Management• Identify Critical Failure• Develop a feasible plan to stabilize customer’s

satisfaction.• Workaround.• Rollback.

04/10/2023Confidential | Copyright 2012 Trend

Micro Inc.33

Quantity Estimation• Goal: Cost Effective Quantity Estimation

• For estimating• For wrong estimation

• What if the estimation is not correct?• Too few• Too many

04/10/2023Confidential | Copyright 2012 Trend

Micro Inc.34

Scalability & Elasticity• Scalability

• Is your application horizontal scalable?

• Elasticity• Speed of commissioning / decommissioning• Max amount of resource can be brought in• Granularity of usage accounting

• Develop the plan for high traffic.

04/10/2023Confidential | Copyright 2012 Trend

Micro Inc.35

Dare to fail• Fail is inevitable for fast changing application. (Web,

service)• Fast changing• Create dare-to-fail process and environment

• Facebook• Backup plan• Rollback plan