urbancode deploy devops best practices
TRANSCRIPT
© 2016 IBM Corporation
UrbanCode Deploy DevOps Best Practices MichaelD.Elder
IBMDistinguished EngineerChiefDeveloper, UrbanCode
[email protected]@mdelder
Get this deckhttp://bit.ly/2c1G8sy
2© 2016 IBM Corporation
PleaseNote
2
• IBM’sstatementsregardingitsplans,directions,andintentaresubjecttochangeorwithdrawalwithoutnoticeatIBM’ssolediscretion.
• Informationregardingpotentialfutureproductsisintendedtooutlineourgeneralproductdirectionanditshouldnotbereliedoninmakingapurchasingdecision.
• Theinformationmentionedregardingpotentialfutureproductsisnotacommitment,promise,orlegalobligationtodeliveranymaterial,codeorfunctionality.Informationaboutpotentialfutureproductsmaynotbeincorporatedintoanycontract.
• Thedevelopment, release, andtimingofanyfuturefeaturesorfunctionalitydescribed forourproductsremainsatoursolediscretion.
• Performance isbasedonmeasurementsandprojectionsusingstandardIBMbenchmarksinacontrolledenvironment.The actualthroughputorperformance thatanyuserwillexperience willvarydependinguponmanyfactors,includingconsiderationssuchastheamountofmultiprogrammingintheuser’sjobstream,theI/Oconfiguration,thestorageconfiguration,andtheworkloadprocessed. Therefore, noassurancecanbegiventhatanindividualuserwillachieveresultssimilartothosestatedhere.
3© 2016 IBM Corporation
•UnderstandingDevOps
•DevOpsBestPractices
• Clouds&Containers
Agenda
IBMConfidential
© 2016 IBM Corporation
UnderstandingDevOps
5© 2016 IBM Corporation
Failures duetoinconsistentdev and
productionenvironments
Bottlenecks tryingtodelivermorefrequentreleases tomeetmarketdemands
Complex,manual,processesforreleaselackrepeatabilityand
speed
Poorvisibilityintodependencies across
releases, resources,andteams
Softwaredeliverychallenges
DailyBuild
MonthlyDelivery
Whodidthis lasttime?
Dave…
Dave’snothereman…
Dev
Prod
6© 2016 IBM Corporation
HighperformingteamsadoptDevOps
Reference: 2013 State of DevOps Report by PuppetLabs
7© 2016 IBM Corporation
Intuitive and Scalable Model Driven Deployment
Composite Applications
Components
Re-usable Workflows Environment ManagementSIT
PROD
The “What” The “How” The “Where”
DEPLOYMENT AUTOMATION
8© 2016 IBM Corporation
Intuitive and Scalable Model Driven Deployment
Composite Applications
Components:Container Images
Re-usable Workflows Environment ManagementSIT
PROD
The “What” The “How” The “Where”
DEPLOYMENT AUTOMATION
$ docker run .. {image}:{version}
9© 2016 IBM Corporation
Aboutphilosophy
Cultureofcontinualexperimentationand
learning
• Productionlikeenvironments• Fullyautomateddeployments• Accelerateddelivery cycles
Build– Measure- Learn
“Successisnotdeliveringafeature; successislearninghowtosolvethecustomer’sproblem.”
10© 2016 IBM Corporation 10
IBMBluemix GarageMethod
PracticesProcesses or actions you & your team perform to achieve an outcome.
TracksA selected set of practices, tools, and toolchains to create a cloud native or cloud enabled solution
ToolchainComplementary & compatible tools that when used together create a flexible solution
Experts & ExperiencesListen to our Experts to learn how we transformed our business
Combines industry bestpracticesforDesignThinking, LeanStartup,AgileDevelopment,DevOps,and Cloudtobuildanddeliverinnovative solutions.
https://www.ibm.com/devops/method
11© 2016 IBM Corporation © IBM Corporation
Newcontentaddedweekly
12© 2016 IBM Corporation
Accelerate Innovation
ivDeliverCodeThink Run
Systems of engagement
Systems of Record
Enterprise Applications
Production
MostenterprisesneedacombinationofDevOpspatterns
12
API Catalog
Operational Excellence
ivDeliverCodeThink Run Production
Shared Services
Digital Applications
• Coordinatingdifferentteamsworkingatdifferentspeeds
• Balancing between agility,stability, andrisk
• DecoupleacrossAPIboundary,butmanagerelease processwithconsistent approach
13© 2016 IBM Corporation
ExpectationsofContinuousDeliveryUser Experience Today Emerging
Primary Workload Type Software supports core business function
Software enables innovative business approaches to differentiate from competitionBig Data, Analytics, Mobile/Social Channels
Delivery Model Planned Incremental (DevOps)
Development and Operations Team Sizes 100s and costly 10s with built-in DevOps automation
Release Frequency Months to years Days to weeks, based on business opportunity
Integration Frequency Weeks Continuous
Infrastructure Deployment Days Minutes
Time to Value Planned Opportunistic
Operational Model Systems management Built into application, recovery-oriented, continuous availability
Service Sourcing Develop Consume and assemble (public and private)
Feedback Loop Periodic Continuous interaction with clients
14© 2016 IBM Corporation
Increasetheoperationalawarenessofyoursoftware earlierinthedevelopmentprocess.
WhatdoesDevOpsmeantome?
Howdoyoudothat?§ Architecture
§ AutomatedTesting
§ AutomatedDeployments
§ Production-LikeEnvironments
§ AutomatedReleasePromotionProcess
§ Versioncontrolofallsoftware,automation,andconfigurationWhydothis?
Becausefasterfeedbackloopsenablerapidevolution ofideasandthereforefasteriterationsofyoursoftware
ShiftLeft
15© 2016 IBM Corporation
Bettercommunication betweenthosewho createandthosewhooperate(samepeopleinsomecases)
Reducedfearofbreakingthebuild/deployment/environment
Failfastbeforeyou“failall”
Heavyfocusonexperimentationandlearning
Improvethespeedofyourfeedbacklooptoenablerapidevolutionofideas
WhatarethecharacteristicsofteamswhopracticeDevOps?
© 2016 IBM Corporation
DevOpsBestPractices
17© 2016 IBM Corporation
§ Architecture
§ AutomatedTesting
§ AutomatedDeployments
§ Production-LikeEnvironments
§ AutomatedReleasePromotionProcess
§ Versioncontrolofallsoftware,automation,andconfiguration
Aboutournexttopics
18© 2016 IBM Corporation
IBMUrbanCode forContinuousDelivery
Drive down costReduce amount of manual labor, resource wait-time, and rework by eliminating errors & providing self-service environments
Speed time to marketIncrease frequency of software delivery through automated, repeatable deployment processes across development, test and production
Reduce riskRobust configuration management, coordinated release processes, audits, and traceability
Enabling clients to continuously deliver applications with high quality and low risk
IBM UrbanCode Build is a continuous integration and build management server optimized for the enterprise, designed to make it easy to scale the configuration and management of build infrastructure and seamlessly plug in to development, testing and release tooling.
IBM UrbanCode Deploy enables hybrid cloud by automating the deployment of applications, databases, configurations and containers into development, test and production on prem or in the cloud, helping to drive down cost, speed time to market with reduced risk. With full-stack environment management and deployment automation, users to design, deploy and update full-stack environments for multiple clouds, including Bluemix, Softlayer, AWS, Azure, and virtualized environments.
IBM UrbanCode Release is an intelligent collaboration release management solution that replaces error-prone manual spreadsheets and streamlines release activities for application and infrastructure changes.
19© 2016 IBM Corporation
Aboutyourarchitecture
• ArchitectureshouldsupportDevOps principlessuchasstagedrollout,operationalinsights,andscriptability
• Eachresourceprovidessomeverypracticaladviceforbuildingsystemswhicharefocusedonreliabilityandfeedbackloops
Experiment!:Websiteconversionrate
optimizationwithA/Bandmultivariatetesting
ReleaseIt!:DesignandDeployProduction-Ready
Software
20© 2016 IBM Corporation
Applicationmodel
Reliablycaptureapplicationcontentandconfigurationsettingswithversioningandtraceability
21© 2016 IBM Corporation
Applicationenvironments
Definewherecomponentsaredeployedandcaptureconfigurationsettingsperdeploymentenvironmentforanapplication
22© 2016 IBM Corporation
Templatingforgoodarchitecture
Templatescanbecreatedtostandardizecomponentprocesses&config
23© 2016 IBM Corporation
Createapplicationtemplatesforacceleratedonboarding
Templatescanbecreatedtostandardizecomponentprocesses&config
24© 2016 IBM Corporation
Automatedtestingiscriticalforautomatedfeedback
• This isTHEHARDESTpartofthewholeexperience
• NOONEisgreatatthis,sodon’tgetdiscouraged
• Facebook’s processforcodereview include statsonhowmanytests areconsistently brokentodecidewhether thetestfailuresarenoteworthy
• Continuously improveyourcoverage, flexibility, anddiscipline
• Consider allaspects oftesting:
• ProgrammaticAPI(JUnit)
• RESTAPI(HTTPUnit, RESTFuse)
• WebUI(Selenium)
• Behaviordriventesting (Cucumber)
• Performancetesting (ApacheJMeter)
• Securitytesting (RationalAppScan &IBMQRadar)
25© 2016 IBM Corporation
Embeddingunittestsinyourclassesdirectly
• PromotedapproachbyNetflixtoreducefrictionandintroduces limited additionalbytes relativetothirdpartylibraries
• Makes iteasytowritecommands,theUnitTestbecomes thetestharnessandverification
• AlwaystestSuccess() andtestFailure() usecases toensureexpected behavior
importstaticorg.junit.Assert.assertEquals;importstaticorg.junit.Assert.assertNotNull;
publicclassOSDeployEnvironmentCommand extendsOSAbstractOrchestrationCommand<Response>{
publicstaticclassUnitTest {@TestpublicvoidtestSuccess() throwsInterruptedException,
ExecutionException,JSONException {…Future<Response>request=
newOSDeployEnvironmentCommand(…).queue();
Responseresponse =request.get();assertNotNull(response);assertEquals(HttpStatus.SC_OK,
response.getStatus());}
}
26© 2016 IBM Corporation
TestingYourDeployedApplication– RESTAPI
• WehaveusedRESTFusewhich isanEclipse project
• Captures RESTcallsasannotations, freeingupyourtestmethodtojustverifybehavior
• Requires alittle“hoopjumping”fortesting aseries ofcalls (e.g.POST..PUT..GET)
importstaticcom.eclipsesource.restfuse.Assert.assertCreated;import...restfuse.annotation.Authentication;import...restfuse.annotation.HttpTest;import...restfuse.MediaType;
@HttpTest (method=com.eclipsesource.restfuse.Method.POST, contentMethod="getBlueprintRequestBody",authentications={@Authentication(type=BASIC,user="{keystoneUser}",password="{keystonePassword}")},path="/rest/blueprint",type=MediaType.APPLICATION_JSON
)publicvoidtestCreateBlueprintOK()throwsException{assertCreated(response);}
27© 2016 IBM Corporation
BehaviorDrivenDevelopment&Testing
• Capturesnatural languagestatements using“Given”,“When”,“Then”clauses
• Eachline ismatched toasnippet ofcodebasedonregularexpression
• Complete codestatementsareexecuted tovalidatebehavior
• Bridgebetween businessgoalsandtechnical details
Scenario:Success– UploadclientdirectoryGivenIprovidevalidcredentialstotheapi clientAndIwanttousetheapi clienttomakerequestsWhenIuploadthedirectory”data/sample-app”withtheclientThenthefilesreturnedmatchestheuploaddirectory”data/sample-app"
28© 2016 IBM Corporation
TestingYourDeployedApplication– WebUI
• Selenium istheonly“apparent”choicethatwe’vecomeacrossthat’seffective
• Couples automated testtosomedetails ofyourwebUIandmustbekept insync
• We’vedeveloped afewbestpractices aroundlocationelements formaximumflexibility, couldpublishthisifthere’sinterest
publicclassEditorsPage extendsUCDPPage {…@FindBy(id="new-blueprint-button")privateWebElement newButton;…publicvoidcreateFile(String name,Stringid,String[]values){newButton.click();util().waitForElementPresentAndVisible(By.className("new-file-dialog"));WebElement widget=util().findElement(By.className("new-file-dialog"));
Dialog dialog=Widget.init(this, widget,Dialog.class);dialog.setValues(values, true,true);dialog.clickPrimaryButton();
…}…}
29© 2016 IBM Corporation
Automateddeployments• Visibility andautomatedcontrolofyourapplication deployment process
• Manageapplicationcomponents andversions
• Manageconfigurationsacrossallenvironments
• Offersecure‘self-service’capabilities
• Increasetransparency• Ensuregovernance andcompliancy
30© 2016 IBM Corporation
Defineautomatedprocesses foreachapplicationcomponentEachcomponentdefinesaseriesofsteps,backedby
plugintasksorcustomscripts,todeployit
31© 2016 IBM Corporation
Defineautomatedprocesses fortheapplicationOrdercomponentsorapplylogicwhichis“outside”of
anyspecificcomponent
32© 2016 IBM Corporation
IBMUrbanCodeDeploy– Plug-ins
Over 150 – in over 50 categories
33© 2016 IBM Corporation
Production-LikeEnvironments– UnifiedResourceTree
Provideslogicalhierarchyofdeployedartifactsandtheirmiddlewarecontainers
DEVResources- WebHost- WebComponent- Database- DatabaseComponent
QAResources- WebHost1- WebComponent- WebHost2- WebComponent- Database- DatabaseComponent
PRODResources- WebHost1- WebComponent- WebHost2- WebComponent- WebHost3- WebComponent- WebHost4- WebComponent- DatabasePrimary- DatabaseComponent- DatabaseSecondary- DatabaseComponent
34© 2016 IBM Corporation
Production-LikeEnvironments– LocalDevelopment
PossiblyoneofthemostimportantprinciplesofDevOpsShiftleftanyoperationalconcernsthatleadto
differencesfromDEVtoPROD
https://hub.docker.com/u/ibmcom/#
35© 2016 IBM Corporation
DeploymentofApplicationsacrossEnvironmentsApplicationsaremanagedfordeployment
Componentsfromtheapp
SpecificversionsoftheComponents’artifactsaredeployed
Compliancykeepstrackofwhethertheactualversionsmatchestheexpected
EachComponentknowsitsrelevantdeploymentprocesses
CompareversionsacrossenvironmentsanddefineSnapshotsofknownworkingsetsofComponents
Viewdeploymentrequestsagainsttheenvironment
https://developer.ibm.com/urbancode/AutomatedReleasePromotionProcess
36© 2016 IBM Corporation
PromotionLifecycles
• Shouldfocusonconsistencyfromendtoend
• Usethesamelogicalorganizationandcomponentautomation inDEV,QA,PRODtovalidatetheautomationearly inthedelivery cycle
• Applyapprovalgateswherenecessary toensureappropriatequalityvalidationhasoccurred
IBMCommerce
WebSphere ND
Static Content
Commerce Data
IBMCommerce
WebSphere ND
Static Content
Commerce Data
37© 2016 IBM Corporation
QualityAssurance– Automatedgates
• UseComponent Statuses andEnvironmentGateswherepossible
• Definestatuses forallApps
• UseRESTAPI/CLItotag/approveComponents orSnapshots basedonautomated verification(CIorothertesting)
• Onlycomponentsversions/snapshots withtagcanenterenvironment
38© 2016 IBM Corporation
QualityAssurance– ManualSignoff
• Manualapprovalsarepossible, butrecommendthatasmuchaspossible isautomated (e.g.withstatuses+gates)
• Approvalprocesscantriggeremails orsome clients use”chatops”tools like Slacktostreamline theprocess
39© 2016 IBM Corporation
Inventoryovertime
40© 2016 IBM Corporation
ConfigurationManagement- SnapshotsCreating a Snapshot
Component Versions
1
1
2
2
3
3
321Web
Mid. Code
DB
Snapshot
3
2
1
Mid. Config 1 2 3 3
Snapshot Deployment
ENVSnapshot
3
2
1
WEB HOST
MID HOST
DB HOST
2
1
1
3
2
X
3 3 1
Contents ofenvironments thatpasstests• containdeployable versionandconfiguration
Deployments arebasedondeltasTheyprovide Automation,Audit,andVisibility
41© 2016 IBM Corporation
VersionapplicationswithSnapshots
Capturetestedcomponentversionsandconfigurationsettingsasasnapshotthatispromotedtootherenvironmentsforbettervisibilityandcontrol
42© 2016 IBM Corporation
Better Release Planning and Management
• Enterprise Calendaring
• Release Process Checklist Templates
• Environment Reservation
Effective Change and Risk Management
• Rational Team Concert integration
• Impact Analysis
Continuous Delivery with Auto-Progression
• IBM UrbanCode Deploy integration
Increased Visibility and Control
• Pipeline View
• Federated Dashboard
• Segment Dependency Graph
UrbanCodeReleaseTransformchaoticreleaseplanningintostreamlinedreleaseevents
44© 2016 IBM Corporation
Codereviewshouldbepartofyourversioningprocess
§ Defineandenforcecodingstandardsearly intheprocess
§ Automatedeployment toproduction-like systems aspartofautomatic verification
§ Eachcommitshould runautomated testingandcodecoverage,andthetoolchainshouldenforcecompliancebeforemergingorpromotingchanges
45© 2016 IBM Corporation
§ Architecture
§ AutomatedTesting
§ AutomatedDeployments
§ Production-LikeEnvironments
§ AutomatedReleasePromotionProcess
§ Versioncontrolofallsoftware,automation,andconfiguration
Aboutthisdeck
[email protected] http://linkedin.com/in/mdelder @mdelder
© 2016 IBM Corporation
Clouds&Containers
Workload styles vary, but not release methodologyTraditional IT Cloud-Enabled Cloud-Native
App Servers
+ +
DatabaseWeb Servers
Applications
App Servers
+ +
DatabaseWeb Servers
Applications ServicesApplicationsServices
Managed Runtimes
Traditional IT Cloud-Enabled IT Cloud-Native ITContinuous IntegrationContinuous delivery to
productionCloud-native runtimes (node.js)
Pipeline per microservice
Change Record
CMDB
Automated build / deploy of VMs & Containers
Stage Gates, Co-ordinated Releases
Manual Change & ReleaseSome automation
Audit reports
CABAssessment
&Approval Theory
Reality
IBM Control DeskBMC Remedy
As a Business Owner, I want my team to deliver as fast as possible, but with appropriate controls when needed.
But don’t slow everything down to the slowest speed when controls are added!
Change Management vs. Continuous Delivery
Accelerate Innovation
ivDeliverCodeThink Run
Systems of engagement
Systems of Record
Enterprise Applications
Production
Most enterprises need a combination of DevOps patterns
49
API Catalog
Operational Excellence
ivDeliverCodeThink Run Production
Shared Services
Digital Applications
§ Coordinating differentteamsworkingatdifferentspeeds
§ Balancing between agility,stability, andrisk
§ Decouple acrossAPIboundary,butmanagerelease processwithconsistent approach
Cloud-Enabled Workloads with UrbanCode Deploy
50
§ Yourapplications aremanagedthroughout theentirerelease processwithpredictable automation andinventoryservices
§ Yourfull-stack blueprints capturetherelationships between yourapplications andcloud infrastructure
§ OpenStackAPIprovidesdefactostandards forinteractingwithallofyourcloudresources
Heat Blueprints
UrbanCodeDeploy
Compute Network Storage
Your Applications & Middleware
Full-stack Application Pattern
51Innovate. Disrupt. Transform. Fast. @Enterprise Scale.Hybrid Cloud Deployments through a Single Point of Control
IBM Bluemix
Cloud Foundry
On-prem IaaS
On-prem Traditional IT
IBMCloudOrchestratorIBMPureApplicationSystem
Manageapplicationdeploymentacrossdev,test,andproductionspanningmultipleclouds
KeyPoints:• Enablefullstackdeployments(OS,patternsand
applications)acrosshybridcloudapplications• EstablishcommonDevOpspipelineswithplugpointsto
supportcontinuous delivery
Virtual Machines
DockerContainers
Other VendorsUrbanCodeDeploy
UrbanCode Release
https://www.youtube.com/watch?v=fFuRg6l3N44
SCREENCAP: Designer & Palette
Watson Health: 6 active teams, more on the way§ SocialCare(AbuDhabi):
– Complex topology, heavy UCD components (2 Gb)– 2 hours roundtrip / provision– 50% Heat engine CPU capacity / provision
§ RealWorldEvidence:
– 4 Offerings
– Analytics workload (DB2 blue, Cognos)§ AnalyticsCore:
– Analytics workload (Spark clusters)– UCDP Rest API driven variable sized cluster size
§ OperationalDashboard:
– Cloud control desk for solution management (provision, monitor, troubleshoot)
§ WatsonHealthCloud:
– Foundational micro-services (data lake, data pipeline, logging, monitoring etc)
53
Abu Dhabi Blueprint
54© 2016 IBM Corporation
Docker is great for developers – everything can be run on your laptop
54
55© 2016 IBM Corporation
Production-LikeEnvironments– LocalDevelopment
PossiblyoneofthemostimportantprinciplesofDevOpsShiftleftanyoperationalconcernsthatleadto
differencesfromDEVtoPROD
https://hub.docker.com/u/ibmcom/#
Continuous Integration for Containers
56
Jenkins
IBM UrbanCode Deploy
Docker TrustedRegistry
Build Image
TrackVersions
Deploy &Promote
§ Jenkins builds Docker images andpublishes them toDocker TrustedRegistry
§ UrbanCodeDeploy immediately becomesawareofnewversions, available fordeployment
§ Schedule deployments, trackreadinessviaenvironment gates,andmanageconfiguration properties securely
§ Deployments toDocker Universal ControlPlane
Docker Integration Architecture
Docker TrustedRegistry
IBM UrbanCode Deploy
swarm
DockerUniversalControlPlane
LegacyDatabase
Legacy Applications
Use UrbanCode Deploy to manage:
Inventory Management Promotion Lifecycle
Separation of Duties Import images from trusted sources
Docker TrustedRegistry
IBM UrbanCode DeployDocker
UniversalControlPlane
LegacyDatabase
Legacy Applications
UrbanCode Deploy & Compose
§ Importyourdocker-compose file
§ Automatically createresourcetreecomponents
§ Updates, Inventorymanagement, Deployments, etc.areallthesame.
59
Dev/Test with Hybrid Cloud
Off-
Prem
On-
Prem
Image Registry
PRODSTAGING
Delivery Pipeline
IBM Bluemixhttps://www.youtube.com/watch?v=Eo1rFsv4BW0
IBM UrbanCode Deploy
© 2016 IBM Corporation
Q&A
Get this deckhttp://bit.ly/2c1G8sy
62© 2016 IBM Corporation62
©Copyright IBMCorporation 2013. All rights reserved. The information contained in these materials is provided for informational purposes only, and isprovided AS ISwithout warranty ofanykind, express orimplied. IBMshall notbe responsible for anydamages arising outof theuse of, orotherwise related to, these materials. Nothing contained in these materials is intended to, nor shall have theeffectof, creatinganywarranties or representations from IBMor its suppliers or licensors, oraltering the terms andconditions of theapplicable license agreement governing theuse of IBMsoftware. References in these materialsto IBMproducts, programs, or services donot imply that theywill beavailable inall countries inwhich IBMoperates. Product release dates and/or capabilities referenced in these materials maychange atanytime atIBM’s sole discretion based onmarketopportunities orother factors, and arenot intended tobe acommitment to future product or feature availability inanyway. IBM, the IBM logo, Rational, theRational logo, Telelogic, theTelelogic logo, andother IBMproducts andservices are trademarks of the International Business Machines Corporation, in the United States, other countries or both.Other company,product, or service names maybe trademarks or service marks ofothers.
www.ibm.com/software