getting hiera and hiera
TRANSCRIPT
![Page 1: Getting Hiera and Hiera](https://reader034.vdocuments.site/reader034/viewer/2022042722/58a4cb861a28ab480e8b5d1d/html5/thumbnails/1.jpg)
Hiera and HieraIntroduction to hiera puppet-camp Melbourne 2015
![Page 2: Getting Hiera and Hiera](https://reader034.vdocuments.site/reader034/viewer/2022042722/58a4cb861a28ab480e8b5d1d/html5/thumbnails/2.jpg)
Who am I
![Page 3: Getting Hiera and Hiera](https://reader034.vdocuments.site/reader034/viewer/2022042722/58a4cb861a28ab480e8b5d1d/html5/thumbnails/3.jpg)
These really were the droids you were looking for
![Page 4: Getting Hiera and Hiera](https://reader034.vdocuments.site/reader034/viewer/2022042722/58a4cb861a28ab480e8b5d1d/html5/thumbnails/4.jpg)
Sportsbet puppet history (according to me)
July 2014 a working puppet code with CICD pipeline < 50 puppetized VM’s
- Mix of physical and virtual servers.- Minimal documentation on how systems were built and configured
Nov 2014 Mobile stack built and configured with puppet.
Feb 2015 %99 percent virtualized
July 2015 ~ 900 SOE machines under puppet management
- ~ 200 machines legacy VM’s to do
![Page 5: Getting Hiera and Hiera](https://reader034.vdocuments.site/reader034/viewer/2022042722/58a4cb861a28ab480e8b5d1d/html5/thumbnails/5.jpg)
AgendaTake this
$database_server = $::environment ? { 'production' => 'proddb1.puppetlabs.vm', 'development' => 'odinsraven.puppetlabs.vm', 'QA' => 'qadb1.puppetlabs.vm', 'staging' => 'stagingdb1.puppetlabs.vm',}
And turn it into this
$database_server = hiera('database_server')
![Page 6: Getting Hiera and Hiera](https://reader034.vdocuments.site/reader034/viewer/2022042722/58a4cb861a28ab480e8b5d1d/html5/thumbnails/6.jpg)
Why Hiera?
“We have a module for that, but I can’t share it because it has all our company-specific data in it.”
![Page 7: Getting Hiera and Hiera](https://reader034.vdocuments.site/reader034/viewer/2022042722/58a4cb861a28ab480e8b5d1d/html5/thumbnails/7.jpg)
So what is it?“Hiera is a key/value lookup tool for configuration data, built to make Puppet better and let you set node-specific data without repeating yourself.”
‘Faster-er. Easier-er. Better-er'
![Page 8: Getting Hiera and Hiera](https://reader034.vdocuments.site/reader034/viewer/2022042722/58a4cb861a28ab480e8b5d1d/html5/thumbnails/8.jpg)
What should I put in Hiera:Business-specific data (i.e. internal NTP server, environment config, etc…)
Sensitive company specific data
Data that you don’t want to share with anyone else
What should I NOT put in Hiera:OS-specific data
Data that EVERYONE ELSE who uses this module will need to know (paths to config files, package names, etc…)
![Page 9: Getting Hiera and Hiera](https://reader034.vdocuments.site/reader034/viewer/2022042722/58a4cb861a28ab480e8b5d1d/html5/thumbnails/9.jpg)
Backend types--:backends: - yaml:hierarchy: - hostname/"%{hostname}" - environment/"%{environment}" - common
:yaml: :datadir: /vagrant/hieradata
yaml , json , file , ldap ,gpg ,eyaml,mysql,postgres, redis
![Page 10: Getting Hiera and Hiera](https://reader034.vdocuments.site/reader034/viewer/2022042722/58a4cb861a28ab480e8b5d1d/html5/thumbnails/10.jpg)
The Hierarchy--:backends: - yaml:hierarchy: - hostname/"%{hostname}" - environment/"%{environment}" - common
:yaml: :datadir: /vagrant/hieradata
Hiera likes facts and interacts with facts
![Page 11: Getting Hiera and Hiera](https://reader034.vdocuments.site/reader034/viewer/2022042722/58a4cb861a28ab480e8b5d1d/html5/thumbnails/11.jpg)
Where the data lives--:backends: - yaml:hierarchy: - hostname/"%{hostname}" - environment/"%{environment}" - common
:yaml: :datadir: /vagrant/hieradata
![Page 12: Getting Hiera and Hiera](https://reader034.vdocuments.site/reader034/viewer/2022042722/58a4cb861a28ab480e8b5d1d/html5/thumbnails/12.jpg)
Complete hiera Config--:backends: - yaml:hierarchy: - hostname/"%{hostname}" - environment/"%{environment}" - common
:yaml: :datadir: /vagrant/hieradata
ln -s /etc/hiera.yaml /etc/puppet/hiera.yaml
You will want to do this to avoid confusion/issues
![Page 13: Getting Hiera and Hiera](https://reader034.vdocuments.site/reader034/viewer/2022042722/58a4cb861a28ab480e8b5d1d/html5/thumbnails/13.jpg)
Hierarchy hell:hierarchy: - "%{clientcert}" - "%{environment}" - "%{osfamily}" - "%{osfamily}/%{operatingsystem}" - "%{osfamily}/%{operatingsystem}/%{os_version_major}" - "%{osfamily}/%{operatingsystem}/%{os_version_minor}" # Repeat until you have 15 levels of WTF
![Page 14: Getting Hiera and Hiera](https://reader034.vdocuments.site/reader034/viewer/2022042722/58a4cb861a28ab480e8b5d1d/html5/thumbnails/14.jpg)
A sample from a real hierarchy :hierarchy: - "virtual/%{::virtual}" - "hostname/%{::fqdn}" - "role/%{::role}/%{calling_module}" - "role/%{::role}" - "datacentre/%{::datacentre}" - "environment/%{::hiera_environment}/%{calling_module}" - "environment/%{::hiera_environment}" - "%{calling_module}" - passwords_fake - common
![Page 15: Getting Hiera and Hiera](https://reader034.vdocuments.site/reader034/viewer/2022042722/58a4cb861a28ab480e8b5d1d/html5/thumbnails/15.jpg)
Use hiera in your modules in 2 waysexplicit:
$foo = hiera('foo', 'default string')
Automatic:
class bar ( $foo = 'default string') {}
called Data Bindings
As of Puppet 3.0.0, Puppet will now do a Hiera lookup for the fully namespaced value of a class parameter
![Page 16: Getting Hiera and Hiera](https://reader034.vdocuments.site/reader034/viewer/2022042722/58a4cb861a28ab480e8b5d1d/html5/thumbnails/16.jpg)
Let's get practical
class democlass ( $param1 = 'Welcome to Puppet Camp') { $param2 = hiera ( 'db_server', 'No DB for you!' ) notice("Running with param1: ${param1} :") notice("Running db server : ${param2} : ")}
puppet apply --modulepath ./puppet/modules puppet/manifests/site.ppNotice: Scope(Class[Democlass]): Running with param1: Welcome to Puppet Camp :Notice: Scope(Class[Democlass]): Running db server : No DB for you! :
class democlass ( $param1 = 'Welcome to Puppet Camp') { $param2 = hiera ( 'db_server', 'No DB for you!' ) notice("Running with param1: ${param1} :") notice("Running db server : ${param2} : ")}
![Page 17: Getting Hiera and Hiera](https://reader034.vdocuments.site/reader034/viewer/2022042722/58a4cb861a28ab480e8b5d1d/html5/thumbnails/17.jpg)
Add some direct hiera lookuppuppet apply --modulepath ./puppet/modules puppet/manifests/site.ppNotice: Scope(Class[Democlass]): Running with param1: Welcome to Puppet Camp :Notice: Scope(Class[Democlass]): Running db server : cinderella :
class democlass ( $param1 = 'Welcome to Puppet Camp') { $param2 = hiera ( 'db_server', 'No DB for you!' ) notice("Running with param1: ${param1} :") notice("Running db server : ${param2} : ")}
cat hieradata/common.yamldb_server: 'cinderella'cat hieradata/common.yamldb_server: 'cinderella'
class democlass ( $param1 = 'Welcome to Puppet Camp') { $param2 = hiera ( 'db_server', 'No DB for you!' ) notice("Running with param1: ${param1} :") notice("Running db server : ${param2} : ")}
![Page 18: Getting Hiera and Hiera](https://reader034.vdocuments.site/reader034/viewer/2022042722/58a4cb861a28ab480e8b5d1d/html5/thumbnails/18.jpg)
Add some hiera data bindingpuppet apply --modulepath ./puppet/modules puppet/manifests/site.ppNotice: Scope(Class[Democlass]): Running with param1: Welcome to the ball :Notice: Scope(Class[Democlass]): Running db server : cinderella :
class democlass ( $param1 = 'Welcome to Puppet Camp') { $param2 = hiera ( 'db_server', 'No DB for you!' ) notice("Running with param1: ${param1} :") notice("Running db server : ${param2} : ")}
cat hieradata/common.yamldemoclass::param1: 'Welcome to the ball'db_server: 'cinderella'
cat hieradata/common.yamldemoclass::param1: 'Welcome to the ball'db_server: 'cinderella'
![Page 19: Getting Hiera and Hiera](https://reader034.vdocuments.site/reader034/viewer/2022042722/58a4cb861a28ab480e8b5d1d/html5/thumbnails/19.jpg)
Detour - facter# facterarchitecture => x86_64augeasversion => 1.0.0blockdevice_sda_model => VBOX HARDDISKblockdevice_sda_size => 21474836480blockdevice_sda_vendor => ATAblockdevices => sdafacterversion => 2.4.4filesystems => ext4,iso9660fqdn => localhost.sportsbet.com.augid => vagranthardwareisa => x86_64hardwaremodel => x86_64hostname => localhostinterfaces => eth0,loipaddress => 10.0.2.15ipaddress_eth0 => 10.0.2.15ipaddress_lo => 127.0.0.1
![Page 20: Getting Hiera and Hiera](https://reader034.vdocuments.site/reader034/viewer/2022042722/58a4cb861a28ab480e8b5d1d/html5/thumbnails/20.jpg)
Add some fact based data bindingpuppet apply --modulepath ./puppet/modules puppet/manifests/site.ppNotice: Scope(Class[Democlass]): Running with param1: Dude where is my slipper :Notice: Scope(Class[Democlass]): Running db server : cinderella :
class democlass ( $param1 = 'Welcome to Puppet Camp') { $param2 = hiera ( 'db_server', 'No DB for you!' ) notice("Running with param1: ${param1} :") notice("Running db server : ${param2} : ")}
cat hieradata/common.yamldemoclass::param1: 'Welcome to the ball'db_server: 'cinderella'
cat hieradata/hostname/localhost.yamldemoclass::param1: 'Dude where is my slipper'
cat hieradata/common.yamldemoclass::param1: 'Welcome to the ball'db_server: 'cinderella'
cat hieradata/hostname/localhost.yamldemoclass::param1: 'Dude where is my slipper'
![Page 21: Getting Hiera and Hiera](https://reader034.vdocuments.site/reader034/viewer/2022042722/58a4cb861a28ab480e8b5d1d/html5/thumbnails/21.jpg)
Add some custom facts puppet apply --modulepath ./puppet/modules puppet/manifests/site.ppNotice: Scope(Class[Democlass]): Running with param1: Dude where is my slipper :Notice: Scope(Class[Democlass]): Running db server : Dr Martin :
class democlass ( $param1 = 'Welcome to Puppet Camp') { $param2 = hiera ( 'db_server', 'No DB for you!' ) notice("Running with param1: ${param1} :") notice("Running db server : ${param2} : ")}
cat hieradata/common.yamldemoclass::param1: 'Welcome to the ball'db_server: 'cinderella'
cat hieradata/hostname/localhost.yamldemoclass::param1: 'Dude where is my slipper'
cat hieradata/environment/dev.yamldemoclass::param1: 'Devs be free'db_server: 'Dr Martin'cat /etc/facter/facts.d/environment.yaml
---environment: dev
cat hieradata/common.yamldemoclass::param1: 'Welcome to the ball'db_server: 'cinderella'
cat hieradata/hostname/localhost.yamldemoclass::param1: 'Dude where is my slipper'
cat hieradata/environment/dev.yamldemoclass::param1: 'Devs be free'db_server: 'Dr Martin'
![Page 22: Getting Hiera and Hiera](https://reader034.vdocuments.site/reader034/viewer/2022042722/58a4cb861a28ab480e8b5d1d/html5/thumbnails/22.jpg)
Importance of hierarchy order:hierarchy: - defaults - "hostname/%{hostname}" - "environment/%{environment}" - common
![Page 23: Getting Hiera and Hiera](https://reader034.vdocuments.site/reader034/viewer/2022042722/58a4cb861a28ab480e8b5d1d/html5/thumbnails/23.jpg)
Add a some custom facts (fixed order) puppet apply --modulepath ./puppet/modules puppet/manifests/site.ppNotice: Scope(Class[Democlass]): Running with param1: Devs be free :Notice: Scope(Class[Democlass]): Running db server : Dr Martin :
class democlass ( $param1 = 'Welcome to Puppet Camp') { $param2 = hiera ( 'db_server', 'No DB for you!' ) notice("Running with param1: ${param1} :") notice("Running db server : ${param2} : ")}
cat hieradata/common.yamldemoclass::param1: 'Welcome to the ball'db_server: 'cinderella'
cat hieradata/hostname/localhost.yaml#democlass::param1: 'Dude where is my slipper'
cat hieradata/environment/dev.yamldemoclass::param1: 'Devs be free'db_server: 'Dr Martin'
cat /etc/facter/facts.d/environment.yaml---environment: dev
cat hieradata/common.yamldemoclass::param1: 'Welcome to the ball'db_server: 'cinderella'
cat hieradata/hostname/localhost.yaml#democlass::param1: 'Dude where is my slipper'
cat hieradata/environment/dev.yamldemoclass::param1: 'Devs be free'db_server: 'Dr Martin'
![Page 24: Getting Hiera and Hiera](https://reader034.vdocuments.site/reader034/viewer/2022042722/58a4cb861a28ab480e8b5d1d/html5/thumbnails/24.jpg)
Detour - additional hiera config:merge_behavior
- native (default)- deep - deeper
![Page 25: Getting Hiera and Hiera](https://reader034.vdocuments.site/reader034/viewer/2022042722/58a4cb861a28ab480e8b5d1d/html5/thumbnails/25.jpg)
Debugging hierahiera -d# hiera -d democlass::param1DEBUG: Sat Nov 14 04:27:29 +0000 2015: Hiera YAML backend startingDEBUG: Sat Nov 14 04:27:29 +0000 2015: Looking up democlass::param1 in YAML backendDEBUG: Sat Nov 14 04:27:29 +0000 2015: Looking for data source commonDEBUG: Sat Nov 14 04:27:29 +0000 2015: Found democlass::param1 in commonWelcome to the ball
# hiera -d db_serverDEBUG: Sat Nov 14 04:27:59 +0000 2015: Hiera YAML backend startingDEBUG: Sat Nov 14 04:27:59 +0000 2015: Looking up db_server in YAML backendDEBUG: Sat Nov 14 04:27:59 +0000 2015: Looking for data source commonDEBUG: Sat Nov 14 04:27:59 +0000 2015: Found db_server in commoncinderella
![Page 26: Getting Hiera and Hiera](https://reader034.vdocuments.site/reader034/viewer/2022042722/58a4cb861a28ab480e8b5d1d/html5/thumbnails/26.jpg)
Debugging hiera passing facthiera -d
# hiera -d democlass::param1 environment=devDEBUG: Sat Nov 14 04:28:19 +0000 2015: Hiera YAML backend startingDEBUG: Sat Nov 14 04:28:19 +0000 2015: Looking up democlass::param1 in YAML backendDEBUG: Sat Nov 14 04:28:19 +0000 2015: Looking for data source environment/devDEBUG: Sat Nov 14 04:28:19 +0000 2015: Found democlass::param1 in environment/devDevs be free
# hiera -d db_server environment=devDEBUG: Sat Nov 14 04:28:59 +0000 2015: Hiera YAML backend startingDEBUG: Sat Nov 14 04:28:59 +0000 2015: Looking up db_server in YAML backendDEBUG: Sat Nov 14 04:28:59 +0000 2015: Looking for data source environment/devDEBUG: Sat Nov 14 04:28:59 +0000 2015: Found db_server in environment/devDr Martin
![Page 27: Getting Hiera and Hiera](https://reader034.vdocuments.site/reader034/viewer/2022042722/58a4cb861a28ab480e8b5d1d/html5/thumbnails/27.jpg)
Debugging hiera with facter facts
# facter --yaml > /tmp/facter.yaml # hiera -d democlass::param1 -y /tmp/facter.yamlDEBUG: Sat Nov 14 04:36:28 +0000 2015: Hiera YAML backend startingDEBUG: Sat Nov 14 04:36:28 +0000 2015: Looking up democlass::param1 in YAML backendDEBUG: Sat Nov 14 04:36:28 +0000 2015: Looking for data source hostname/localhostDEBUG: Sat Nov 14 04:36:28 +0000 2015: Found democlass::param1 in hostname/localhostDude where is my slipper
![Page 28: Getting Hiera and Hiera](https://reader034.vdocuments.site/reader034/viewer/2022042722/58a4cb861a28ab480e8b5d1d/html5/thumbnails/28.jpg)
Keeping secretsRecommend using hiera-eyaml (encrypted yaml)
If you use the ATOM editor
- https://atom.io/packages/hiera-eyaml
--- password: > ENC[PKCS7,MIIBeQYJKoZIhvcNAQcDoIIBajCCAWYCAQAxggEhMIIBHQIBAD AFMAACAQEwDQYJKoZIhvcNAQEBBQAEggEACvmEx6+xIWPpykTAsoFyCHb0MC +BWQPsWSm6yeJRgkWV+ZVhWwqz09nS1LMo6aKInzg2EjsDScpUFKeXvgxvV8 IF9Bes33V5ySwnlDR15UAWWkfyYxpjk8ozSTGyIWATRITSMWOZDzOZRBYFcD Mj57o5kykkurVqskYG9DDQ8xPq85t2pPPHjb0d/BzJaeqqXsnacDrylbLJau +Ohldb+s/ekBwj/iDIu2NL3KvRvn2VxYAXrgehi9VXFutN6E8yoE72XkSxfN gBdUNtha3DF9jOjdTx3b43WesqNfAwyFvZ2IRCUzV4K1ZvIDK2pA8bsQDbuP XkX7r2fSgG4CUK4jA8BgkqhkiG9w0BBwEwHQYJYIZIAWUDBAEqBBBo6hA3GV eKTzUgflmH0h9EgBCBe7XgD7ZVLBu3j6YhgL/I]
![Page 29: Getting Hiera and Hiera](https://reader034.vdocuments.site/reader034/viewer/2022042722/58a4cb861a28ab480e8b5d1d/html5/thumbnails/29.jpg)
multiple Backends types --:backends: - eyaml - yaml:hierarchy: - hostname/"%{hostname}" - environment/"%{environment}" - common
:yaml: :datadir: /vagrant/hieradata
![Page 30: Getting Hiera and Hiera](https://reader034.vdocuments.site/reader034/viewer/2022042722/58a4cb861a28ab480e8b5d1d/html5/thumbnails/30.jpg)
Issues you are going to haveComplicated Hierarchy
Latency / Load
Too many backends
Design / Architecture
![Page 31: Getting Hiera and Hiera](https://reader034.vdocuments.site/reader034/viewer/2022042722/58a4cb861a28ab480e8b5d1d/html5/thumbnails/31.jpg)
Unhelpful errorsError: Could not retrieve catalog from remote server: Error 400 on SERVER: (<unknown>): found character that cannot start any token while scanning for the next token at line 1297 column 3Warning: Not using cache on failed catalogError: Could not retrieve catalog; skipping run
Error: Could not retrieve catalog from remote server: Error 400 on SERVER: undefined method `empty?' for nil:NilClass at /etc/puppet/environments/production/manifests/nodes.pp:1 on node example.makandra.deWarning: Not using cache on failed catalogError: Could not retrieve catalog; skipping run
![Page 32: Getting Hiera and Hiera](https://reader034.vdocuments.site/reader034/viewer/2022042722/58a4cb861a28ab480e8b5d1d/html5/thumbnails/32.jpg)
Tips to deal with Unhelpful errorsTypical malformed yaml
Check your yaml with a linter
# cd /vagrant/hieradata/; \for i in `find . -name '*.yaml'`; \ do echo $i; \ruby -e "require 'yaml'; \YAML.parse(File.open('$i'))"; done
![Page 33: Getting Hiera and Hiera](https://reader034.vdocuments.site/reader034/viewer/2022042722/58a4cb861a28ab480e8b5d1d/html5/thumbnails/33.jpg)
The PROsSeparation of data and code
keeping secrets
Backend integration to existing data stores
Make’s some conditional logic disappear
Make’s for better puppet code
![Page 34: Getting Hiera and Hiera](https://reader034.vdocuments.site/reader034/viewer/2022042722/58a4cb861a28ab480e8b5d1d/html5/thumbnails/34.jpg)
The CON’sHard to figure out where things come from
hiera-yaml can only support one datadirectory
debugging
public modules and hiera still has some ways to go
![Page 35: Getting Hiera and Hiera](https://reader034.vdocuments.site/reader034/viewer/2022042722/58a4cb861a28ab480e8b5d1d/html5/thumbnails/35.jpg)
Wrap upUse hiera it’s awesome
Start simple use yaml backend
Have defaults to your lookups for when lookups fail
Experiment with it
Lint check you yaml
![Page 36: Getting Hiera and Hiera](https://reader034.vdocuments.site/reader034/viewer/2022042722/58a4cb861a28ab480e8b5d1d/html5/thumbnails/36.jpg)
Things you might want to look at laterMore Debugging
- Glowing Octo Ninja - https://github.com/llowder/Zwiebelschaler
hiera_array
hiera_hash
hiera_include
create_resources
![Page 37: Getting Hiera and Hiera](https://reader034.vdocuments.site/reader034/viewer/2022042722/58a4cb861a28ab480e8b5d1d/html5/thumbnails/37.jpg)
Please go and upvote this bugPuppet user resource should read only from local databases
https://tickets.puppetlabs.com/browse/PUP-1913
@bigAl