KATELLO AND ANSIBLEKATELLO AND ANSIBLEFOR AUTOMATEDFOR AUTOMATED

TESTING ANDTESTING ANDRELEASING OFRELEASING OF

PACKAGESPACKAGES1

$ WHOAMI$ WHOAMI

Evgeni Golov

Software Engineer at Red Hat

ex-Consultant at Red Hat

Debian and Grml Developer

♥ FOSS ♥

♥ automation ♥

2

MOTIVATIONMOTIVATIONyou build a software productyou ship the product as distribution packagesto your customersthe product has dependencies outside a baseOS (Ruby? node.js? Django?)unit tests are great, but you also need to testthe shipped bits

3

ANSIBLEANSIBLE

4

$ WHATIS ANSIBLE$ WHATIS ANSIBLE

radically simple IT automation enginecontains a big number of modules to executeactions and ensure state on target hostseasily extended by self-written modulesintegrates well with REST APIs

5

ANSIBLE TERMINOLOGYANSIBLE TERMINOLOGYModule - discrete units of code that can beused from the command line or in a playbooktask to execute an action or ensure a stateTask - Module invocation with a set ofparametersPlay - list of Tasks to be executed against a setof hostsPlaybook - �le containing one or more Plays

6

KATELLOKATELLO

7

$ WHATIS KATELLO$ WHATIS KATELLO

plug-in to Foremanadds content management functionality (RPM,DEB, Puppet, Containers, Files)allows to group content for tailoredpresentation to consumersallows snapshots of content for versioning

8

KATELLO TERMINOLOGYKATELLO TERMINOLOGYRepository - Collection of contentProduct - Collection of related repositories(CentOS 7 distribution with repositories fori686 and x86_64)Lifecycle Environment - Environment/stagein your deployment cycle (Test, QA,Production)

Library - special LE that receives the content�rst

9

KATELLO TERMINOLOGYKATELLO TERMINOLOGYContent View - Selection of repositories(CentOS 7 + EPEL 7)

Publish creates a snapshot (Version) of theselected repositories available to LibraryPromote copies a published Content ViewVersion to another LE

Composite Content View - Selection ofContent Views (base OS + Application)

can be published and promoted like a CV

10

KATELLO EXAMPLEKATELLO EXAMPLE

11

STAGING CHANGES WITHSTAGING CHANGES WITHKATELLOKATELLO

every time a (Composite) Content View ispublished, a new Version is createdthis version can be made available to clients bypromoting it to a certain Lifecycle Environmentyou can revert to older versions, if problemsare found after a promotion

12

STAGING CHANGES WITHSTAGING CHANGES WITHKATELLO (EXAMPLE)KATELLO (EXAMPLE)

DEV moving fast, getting changes on everycommitTEST getting changes daily, after a minimalgating happenedQA getting changes weekly, after a basic set oftests passedPROD getting changes whenever QA is happy

13

TESTING WITHTESTING WITHKATELLO AND ANSIBLEKATELLO AND ANSIBLE

14

ARCHITECTURE OVERVIEWARCHITECTURE OVERVIEWSource in Git (GitLab)Jenkins is the main executor, triggered byGitLabKatello is the package storeAnsible is used by Jenkins to interact with theKatello API

15

TEST WORKFLOWTEST WORKFLOWJenkins builds packages on every change(using Koji)Packages are synced to KatelloKatello also syncs external packages (RHEL,RHSCL)Jenkins creates/updates ContentView (RHEL,RHSCL, Packages from Koji)Jenkins tests the content in Library by installingthe software and running end-to-end testsJenkins promotes ContentView to Test and QA

16

PACKAGE BUILDINGPACKAGE BUILDINGOn every change to the source, the following

steps are executed:a new source tarball is generatedthe RPM .spec is updatedthe RPM is built using Koji

17

PACKAGE TESTINGPACKAGE TESTINGJenkins runs a daily pipeline which:

Synchronizes the packages from Koji intoKatello (Library)Executes a test Ansible playbook in a VagrantVMWhen the playbooks �nishes successfully, theContent is promoted to Test

18

PACKAGE TESTINGPACKAGE TESTINGWe use for testingSet of Ansible playbooks and Vagrant �les

Create Vagrant VMsCon�gure package sourcesInstall Katello and a Content ProxyExecute bats tests that verify thefunctionality of the setup

Same setup can be used on your laptop (if ithas enough RAM)

forklift

19

PACKAGE TESTINGPACKAGE TESTINGSynchronization is executed viakatello_sync from

Content View is published viakatello_content_view_publish

Promotion happens viakatello_content_view_version_promote

foreman-ansible-modules

20

FURTHER TESTING ANDFURTHER TESTING ANDRELEASINGRELEASING

Daily tests are limited and take "only" ~1hOnce a week content from Test is promoted toQAThis triggers a large test-suite (>24h!)Plus manual veri�cation of features and �xedbugs that have no automated testsAfter successful veri�cation, the software isreleased

21

ARCHIVING RELEASESARCHIVING RELEASESEach weekly snapshot is archived

to an own Lifecycle Environment (createdwith katello_lifecycle_environment)referenced by an own Activation Key(created with katello_activation_key)

this allows to reproduce older environmentsand re-test bugs

22

REFERENCESREFERENCESour Jenkins jobsour Ansible playbooks

23

THANKS!THANKS!

evgeni@golov.de

die-welt.net

@zhenech

@zhenech@chaos.social

@evgeni

zhenech

24