Krane,Devops andpeopleAndrewYongjoon KongCloudComputingCell

Infra&DataplatformteamKakaocorp

Kakaocorp

KakaocorpAndrew.Yongjoon kong

• CloudTechnicalAdvisoryforGovernmentBroadCastAgency• AdjunctProf.Ajou Univ• KoreaDataBaseAgencyActingProfessorforBigdata• MemberofNationalInformationAgencyBigdata Advisorycommittee• Kakaoà Daum Kakaoà Kakaocorp,CloudComputingCelllead

KoreanEdition

KakaocorpSomeNumbersaboutkakao openstack

1563tenants632 pull request since 2014.9

251 VMs are created/deleted per day

8703 vms

KakaocorpSomeinformationaboutkakao openstack

openstack releasefromgrizzlytoKilo5timesupgraded

total 4Regionpg,seoulv2,seoulv1,busan

additional service Heat/Trove/Sahara

KakaocorpCloud,wedohaveSDN,nothaveOpenflow, Noothers

eth0

Compute node

nova-compute

neutron-linuxbridge-agent

neutron-dhcp-agent

Gateway10.10.100.1

linux bridge

vm

IP:10.10.100.2/32

Routing Table

1 10.100.10.2/32 via 192.1.1.201

BGP

192.1.1.202 BGPVirtual Switch block

Virtual Process block

Neutron-l3-agent

vlan

vlan

Virtual RouterService Route Table

1

Management Route Table1

PracticeFrugalitytoBoostCreativity

KakaocorpSPECIALNetworkResource

• Generally,Knownas/32network.

• NoL2(link)considerationneededanymore(nosubnet)• WithDynamicRoutingProtocol,itmoveeverywhere.• SimpleIPplanning(JustthinkofIPranges)• It’sveryAtomicResource,itkeepsitsIPaftermigrationthroughzones

10.0.0.1 / 32 or IP 10.0.0.1 netmask 255.255.255.255

KakaocorpIntegratedInformationServiceBus&EIP:CodeNameKEMI

Enterprise Integration • Topic based Data ETL• Can cover every computing res

ource ( Physical Server, Virtual instance, Container, Public Cloud )

• Abstracting “Data Center Information layer”

• Can make deep engineering experience over every resources.

PhysicalServers

VirtualInstances Containers External

Clouds

Others(switches,

logs)

monitoring

KEMIIMS

(kakao CMDBAPI)

ESB

RuleEngine

Notification ETL

Data Center Information abstraction layer

API

predicting

scheduling

OpenstackHeat

OtherServiceAPI

DataCenter(orService)ManagementActivity

control

KakaocorpKEMI-stats:The packagesandworkflow

KakaocorpKEMI-log

KakaocorpCITE

KakaocorpDKOS

KakaocorpTheProblem

TheTerm:Devops

• Devops isnolongercompositeoftwowords“Development+Operation”

• Culturalmovementthatchanges• howindividualsthinkabouttheirwork,• valuesthediversityofworkdone,• supportsintentionalprocessesthatacceleratetheratebywhichbusinessesrealizevalue,andmeasurestheeffectofsocialandtechnicalchange

• Devops isaboutfindingwaystoadaptandinnovatesocialstructure,culture,andtechnologytogetherinordertoworkmoreeffectively

SoftwareDevelopmentMethodologies• Waterfall

• Focusonsequentialprogression• Largeamountoftimespentinrequirementsanddesignphase.

SoftwareDevelopmentMethodologies• Agile

• lightweightandflexiblethanpreviousmethodssuchaswaterfall

• AgileManifesto• Weareuncoveringbetterwaysofdevelopingsoftwarebydoingitandhelpingothersdoit.Throughthisworkwehavecometovalue:

• individualsandinteractions overprocessesandtools• workingsoftware overcomprehensivedocumentation• customercollaboration overcontractnegotiation• respondingtochange overfollowingaplan

SoftwareDevelopmentMethodologies• Scrum

• KenSchwaber andDr.JeffSutherlandpresents• focusesonmaximizingadevelopmentteam’sabilitytoquicklyrespondtochangesinbothprojectandcustomerrequirements.

• Sprints(1~4weeks )• Dailystandup• Restrospect

• Whathappened• Whatwentwell/poorly ?

Devops Misconceptions

• Devops onlyforDevelopersandSystemadministrators• Devops isateam• Devops isaJobTitle• Devops MeansDoingAlltheWorkwithHalfthePeople• Devops isaboutTool/Automation

Devops Anti-Patterns

• BlameCulture• Prefernottoblame

• Silos• Prefershareorusableproduct

• RootCauseAnalysis• Preferquick/frequentresponse

Devops forPillars

Collaboration Affinity Tools Scailing

Collaborationcomprises

• Asynchronouscodereview• Documentation• Updatingissuesandbugreport• Demonstratingweeklyprogress• Regularstatusupdates• Pairing

Collaboration:Kakao cloud

• Asynchronouscodereview:Github PullRequest• Documentation:SourcecodeorWikiPage• Updatingissuesandbugreport:

• JiraComment• Githup andJiraIntegration

• Demonstratingweeklyprogress:SprintReview• Regularstatusupdates:DailyStandup, AutoTesting• Pairing:Github Review/Merge

Collaboration:kakao cloud

Collaboration:kakao cloud

• DevelopGithub proxy.• SendingGithup eventtoKakao talk

Affinity

• Havetothinkabout“WhatcomprisesTeam”• HavetofightagainstDiversity

• DiversityinGerder/Race/Nationality/Age/VeteranStatus/Religion/FamilialStatusisOKandinevitable

• WayofHandlingworks• Measurementofsuccess• Communicationstyleinworks• Leadership

• Thekeyis“Fromindividiual toTeams”

Tools

LocalDevelopmentEnvironment VersionControl

Artifact Management Automated Test

DevelopmentTools

Tools: kakao cloud

LocalDevelopmentEnvironmentKFIELD

VersionControlGithub

Artifact ManagementChef +Cookbookversion

Automated TestJenkins’ Github PRTrigger+

Rake+Kitchen +(openstack/docker)

DevelopmentTools

Tools:LocalDevelopmentEnvironment,KField

Tools: VersionControl,Github

• PullRequestbasedcommithistorymanagement• NoSelfMergeruleApplied

ArtifactManagement:Chef/Cookbookversion• TwoSpeciessourcecode

• OneisOpenstack Code(nova,neutron,cinder…..)• OneisOpenstackDeployingCode(chef,infra-automation)

• FromChefcode,itusesopenstack’s certainbranch• Usecookbookmetadataforartifactversion.

• AutomaticEngagedaftertestingandmerged

AutomatedTest&Build

• 3category• OnDemandAutomatedtestandbuild• ScheduledTestandbuild• TriggeredAutomatedTestandBuild

• Morecategory• Smoketesting• RegressionTesting• UsabilityTesting• A/BTesting

AutomatedTest&Build:JenkinsTest• Triggered

• Sometimesuseon-demandtest,“Testthisplease”commentongithub pullrequest

AutomatedTest&Build:JenkinsTest

Toolsisaccelerator(ingood/badway)

• Toolisaboutwhichprogramminglanguages,IDEs,texteditor,shell,configurationmanagementsolution,orchatprogramtouse

• StandardizationofTools• Improvecommunication• Setboundaries• Repairunderstandingwithinthescopeofthedevops compact

• ToolsImpactingCommunication• Toolcanreduce(oradd)thefriction

References

• Effectivedevops (O’reilly)• TestDriveninfrastructurewithchef(O’reilly)• Infrastructureasacode(O’reilly)• 클린코드 CleanCode( Acornpub)• 리눅스네트워크의이해 UnderstandingLinuxnetworkinternals (Acornpub)

• PersonalExperience(andrew,justjoke)