growing pains: avoiding module hell
DESCRIPTION
As the Puppet community grows by leaps and bounds we find ourselves having many of the same problems other software development communities have. Whether its libraries in C, JARs in Java, or gems in Ruby the basic unit of software can be a huge source of problems. With the proliferation of modules on the forge, Github, or internally grow we face the same problems. How many times have you gone to pull in a module only to find it requires a conflicting sudo module? How many Apache modules are there and why are they all different? Where did that class go, the version only changed by 0.0.1? As a community we should address these challenges early or face our own module hell. Chad Metcalf Infrastructure Engineer, WibiData Chad Metcalf leads the Infrastructure Engineering team at WibiData. His team is charged with developing the infrastructure, tools and automation that help drive a company building Big Data Applications. Previously he lead the Infrastructure Operations team at Cloudera building large Hadoop clusters and occasionally pretending to be a network engineer. When not pretending to be someone who understands networking he enjoys living in San Francisco, taking his wife out for dinner, and playing with his 75 pound pit-mix, River.TRANSCRIPT
![Page 1: Growing Pains: Avoiding Module Hell](https://reader034.vdocuments.site/reader034/viewer/2022042518/5595a1801a28ab00448b4768/html5/thumbnails/1.jpg)
Growing Pains: Avoiding Module
Hell Chad Metcalf
@metcalfc
Saturday, August 24, 13
![Page 2: Growing Pains: Avoiding Module Hell](https://reader034.vdocuments.site/reader034/viewer/2022042518/5595a1801a28ab00448b4768/html5/thumbnails/2.jpg)
Saturday, August 24, 13
![Page 3: Growing Pains: Avoiding Module Hell](https://reader034.vdocuments.site/reader034/viewer/2022042518/5595a1801a28ab00448b4768/html5/thumbnails/3.jpg)
Saturday, August 24, 13
![Page 4: Growing Pains: Avoiding Module Hell](https://reader034.vdocuments.site/reader034/viewer/2022042518/5595a1801a28ab00448b4768/html5/thumbnails/4.jpg)
Saturday, August 24, 13
![Page 5: Growing Pains: Avoiding Module Hell](https://reader034.vdocuments.site/reader034/viewer/2022042518/5595a1801a28ab00448b4768/html5/thumbnails/5.jpg)
As do many stories, it all begins with ...
Saturday, August 24, 13
![Page 6: Growing Pains: Avoiding Module Hell](https://reader034.vdocuments.site/reader034/viewer/2022042518/5595a1801a28ab00448b4768/html5/thumbnails/6.jpg)
$ sudo
Saturday, August 24, 13
![Page 7: Growing Pains: Avoiding Module Hell](https://reader034.vdocuments.site/reader034/viewer/2022042518/5595a1801a28ab00448b4768/html5/thumbnails/7.jpg)
Saturday, August 24, 13
![Page 8: Growing Pains: Avoiding Module Hell](https://reader034.vdocuments.site/reader034/viewer/2022042518/5595a1801a28ab00448b4768/html5/thumbnails/8.jpg)
Saturday, August 24, 13
![Page 9: Growing Pains: Avoiding Module Hell](https://reader034.vdocuments.site/reader034/viewer/2022042518/5595a1801a28ab00448b4768/html5/thumbnails/9.jpg)
sudo::conf { 'web': source => 'puppet:///files/etc/sudoers.d/web', } sudo::conf { 'admins': priority => 10, content => "%admins ALL=(ALL) NOPASSWD: ALL\n", } sudo::conf { 'joe': priority => 60, source => 'puppet:///files/etc/sudoers.d/users/joed', }
Saturday, August 24, 13
![Page 10: Growing Pains: Avoiding Module Hell](https://reader034.vdocuments.site/reader034/viewer/2022042518/5595a1801a28ab00448b4768/html5/thumbnails/10.jpg)
sudo::rule { "extra_rule": ensure => present, who => 'bob', commands => "/usr/sbin/systemctl", nopass => false, comment => "what ever you like", }
Saturday, August 24, 13
![Page 11: Growing Pains: Avoiding Module Hell](https://reader034.vdocuments.site/reader034/viewer/2022042518/5595a1801a28ab00448b4768/html5/thumbnails/11.jpg)
Saturday, August 24, 13
![Page 12: Growing Pains: Avoiding Module Hell](https://reader034.vdocuments.site/reader034/viewer/2022042518/5595a1801a28ab00448b4768/html5/thumbnails/12.jpg)
Infrastructure as Code
Module as an API
Source Code Management
Verification and Validation
Release Management
Saturday, August 24, 13
![Page 13: Growing Pains: Avoiding Module Hell](https://reader034.vdocuments.site/reader034/viewer/2022042518/5595a1801a28ab00448b4768/html5/thumbnails/13.jpg)
Saturday, August 24, 13
![Page 14: Growing Pains: Avoiding Module Hell](https://reader034.vdocuments.site/reader034/viewer/2022042518/5595a1801a28ab00448b4768/html5/thumbnails/14.jpg)
Saturday, August 24, 13
![Page 15: Growing Pains: Avoiding Module Hell](https://reader034.vdocuments.site/reader034/viewer/2022042518/5595a1801a28ab00448b4768/html5/thumbnails/15.jpg)
Saturday, August 24, 13
![Page 16: Growing Pains: Avoiding Module Hell](https://reader034.vdocuments.site/reader034/viewer/2022042518/5595a1801a28ab00448b4768/html5/thumbnails/16.jpg)
Saturday, August 24, 13
![Page 17: Growing Pains: Avoiding Module Hell](https://reader034.vdocuments.site/reader034/viewer/2022042518/5595a1801a28ab00448b4768/html5/thumbnails/17.jpg)
Saturday, August 24, 13
![Page 18: Growing Pains: Avoiding Module Hell](https://reader034.vdocuments.site/reader034/viewer/2022042518/5595a1801a28ab00448b4768/html5/thumbnails/18.jpg)
Saturday, August 24, 13
![Page 19: Growing Pains: Avoiding Module Hell](https://reader034.vdocuments.site/reader034/viewer/2022042518/5595a1801a28ab00448b4768/html5/thumbnails/19.jpg)
Saturday, August 24, 13
![Page 20: Growing Pains: Avoiding Module Hell](https://reader034.vdocuments.site/reader034/viewer/2022042518/5595a1801a28ab00448b4768/html5/thumbnails/20.jpg)
Saturday, August 24, 13
![Page 21: Growing Pains: Avoiding Module Hell](https://reader034.vdocuments.site/reader034/viewer/2022042518/5595a1801a28ab00448b4768/html5/thumbnails/21.jpg)
Saturday, August 24, 13
![Page 22: Growing Pains: Avoiding Module Hell](https://reader034.vdocuments.site/reader034/viewer/2022042518/5595a1801a28ab00448b4768/html5/thumbnails/22.jpg)
Saturday, August 24, 13
![Page 23: Growing Pains: Avoiding Module Hell](https://reader034.vdocuments.site/reader034/viewer/2022042518/5595a1801a28ab00448b4768/html5/thumbnails/23.jpg)
Saturday, August 24, 13
![Page 24: Growing Pains: Avoiding Module Hell](https://reader034.vdocuments.site/reader034/viewer/2022042518/5595a1801a28ab00448b4768/html5/thumbnails/24.jpg)
semver.org
Saturday, August 24, 13
![Page 25: Growing Pains: Avoiding Module Hell](https://reader034.vdocuments.site/reader034/viewer/2022042518/5595a1801a28ab00448b4768/html5/thumbnails/25.jpg)
MAJOR.MINOR.PATCH
Saturday, August 24, 13
![Page 26: Growing Pains: Avoiding Module Hell](https://reader034.vdocuments.site/reader034/viewer/2022042518/5595a1801a28ab00448b4768/html5/thumbnails/26.jpg)
MAJOR version when you make incompatible API changes
1.x.x
Saturday, August 24, 13
![Page 27: Growing Pains: Avoiding Module Hell](https://reader034.vdocuments.site/reader034/viewer/2022042518/5595a1801a28ab00448b4768/html5/thumbnails/27.jpg)
MINOR version when you add functionality in a backwards-compatible manner
x.0.x
Saturday, August 24, 13
![Page 28: Growing Pains: Avoiding Module Hell](https://reader034.vdocuments.site/reader034/viewer/2022042518/5595a1801a28ab00448b4768/html5/thumbnails/28.jpg)
PATCH version when you make backwards-compatible bug fixes
x.x.0
Saturday, August 24, 13
![Page 29: Growing Pains: Avoiding Module Hell](https://reader034.vdocuments.site/reader034/viewer/2022042518/5595a1801a28ab00448b4768/html5/thumbnails/29.jpg)
Its OK to go:
1.0.0
Saturday, August 24, 13
![Page 30: Growing Pains: Avoiding Module Hell](https://reader034.vdocuments.site/reader034/viewer/2022042518/5595a1801a28ab00448b4768/html5/thumbnails/30.jpg)
Saturday, August 24, 13
![Page 31: Growing Pains: Avoiding Module Hell](https://reader034.vdocuments.site/reader034/viewer/2022042518/5595a1801a28ab00448b4768/html5/thumbnails/31.jpg)
Easy to learnEasy to useHard to misuseEasy to
Saturday, August 24, 13
![Page 32: Growing Pains: Avoiding Module Hell](https://reader034.vdocuments.site/reader034/viewer/2022042518/5595a1801a28ab00448b4768/html5/thumbnails/32.jpg)
Gather requirementsUse casesDon’t over specifyPrivate as possible
Saturday, August 24, 13
![Page 33: Growing Pains: Avoiding Module Hell](https://reader034.vdocuments.site/reader034/viewer/2022042518/5595a1801a28ab00448b4768/html5/thumbnails/33.jpg)
Saturday, August 24, 13
![Page 34: Growing Pains: Avoiding Module Hell](https://reader034.vdocuments.site/reader034/viewer/2022042518/5595a1801a28ab00448b4768/html5/thumbnails/34.jpg)
Saturday, August 24, 13
![Page 35: Growing Pains: Avoiding Module Hell](https://reader034.vdocuments.site/reader034/viewer/2022042518/5595a1801a28ab00448b4768/html5/thumbnails/35.jpg)
Saturday, August 24, 13
![Page 36: Growing Pains: Avoiding Module Hell](https://reader034.vdocuments.site/reader034/viewer/2022042518/5595a1801a28ab00448b4768/html5/thumbnails/36.jpg)
Saturday, August 24, 13
![Page 37: Growing Pains: Avoiding Module Hell](https://reader034.vdocuments.site/reader034/viewer/2022042518/5595a1801a28ab00448b4768/html5/thumbnails/37.jpg)
Nifty
Nifty::Firewall
Nifty::Monitoring
Nifty::Supervisor
iptables
Icinga
Monit
Firewall::Nifty
Monitoring::Nifty
Supervisor::Nifty
Saturday, August 24, 13
![Page 38: Growing Pains: Avoiding Module Hell](https://reader034.vdocuments.site/reader034/viewer/2022042518/5595a1801a28ab00448b4768/html5/thumbnails/38.jpg)
Nifty
Nifty::Firewall
Nifty::Monitoring
Nifty::Supervisor
pf
Sensu
Supervisord
Firewall::Nifty
Monitoring::Nifty
Supervisor::Nifty
Saturday, August 24, 13
![Page 39: Growing Pains: Avoiding Module Hell](https://reader034.vdocuments.site/reader034/viewer/2022042518/5595a1801a28ab00448b4768/html5/thumbnails/39.jpg)
Saturday, August 24, 13
![Page 40: Growing Pains: Avoiding Module Hell](https://reader034.vdocuments.site/reader034/viewer/2022042518/5595a1801a28ab00448b4768/html5/thumbnails/40.jpg)
Don’t Specify ALL THE THINGS!!!
Saturday, August 24, 13
![Page 41: Growing Pains: Avoiding Module Hell](https://reader034.vdocuments.site/reader034/viewer/2022042518/5595a1801a28ab00448b4768/html5/thumbnails/41.jpg)
define apache::vhost ( $server_admin $server_name = '', $docroot = '', $docroot_create = false, $docroot_owner = 'root', $docroot_group = 'root', $port = '80', $ssl = false, $template = 'apache/virtualhost/vhost.conf.erb', $priority = '50', $serveraliases = '', $env_variables = '', $passenger = false, $passenger_high_performance = true, $passenger_max_pool_size = 12, $passenger_pool_idle_time = 1200, $passenger_max_requests = 0, $passenger_stat_throttle_rate = 30, $passenger_rack_auto_detect = true, $passenger_rails_auto_detect = false, $passenger_rails_env = '',23 $passenger_rails_base_uri = '',24 $passenger_rack_env = '',25 $passenger_rack_base_uri = '',26 $enable = true,27 $directory = '',28 $directory_options = '',29 $directory_allow_override = 'None'30 )
Saturday, August 24, 13
![Page 42: Growing Pains: Avoiding Module Hell](https://reader034.vdocuments.site/reader034/viewer/2022042518/5595a1801a28ab00448b4768/html5/thumbnails/42.jpg)
<?xml version="1.0" encoding="UTF-8"?> 2 3 <!--Autogenerated by Cloudera CM on 2013-08-07T23:41:06.864Z--> 4 <configuration> 5 <property> 6 <name>mapred.job.tracker</name> 7 <value>xwing07.ul.wibidata.net:8021</value> 8 </property> 9 <property>10 <name>mapreduce.job.counters.max</name>11 <value>120</value>12 </property>13 <property>14 <name>mapred.output.compress</name>15 <value>false</value>16 </property>17 <property>18 <name>mapred.output.compression.type</name>19 <value>BLOCK</value>20 </property>21 <property>22 <name>mapred.output.compression.codec</name>23 <value>org.apache.hadoop.io.compress.DefaultCodec</value>24 </property> <property> <name>mapred.map.output.compression.codec</name> <value>org.apache.hadoop.io.compress.SnappyCodec</value> </property> <property> <name>mapred.compress.map.output</name> <value>true</value> </property> <property> <name>io.sort.factor</name> <value>64</value> </property> <property> <name>io.sort.record.percent</name> <value>0.05</value> </property> <property> <name>io.sort.spill.percent</name> <value>0.8</value> </property> <property> <name>mapred.reduce.parallel.copies</name>47 <value>10</value>48 </property>49 <property>50 <name>mapred.submit.replication</name>51 <value>2</value>52 </property>53 <property>54 <name>mapred.reduce.tasks</name>55 <value>24</value>56 </property>57 <property>58 <name>io.sort.mb</name>59 <value>233</value>60 </property>61 <property>62 <name>mapred.child.java.opts</name>63 <value> -Xmx978972870</value>64 </property>65 <property>66 <name>mapred.job.reuse.jvm.num.tasks</name>67 <value>1</value>68 </property>69 <property>70 <name>mapred.map.tasks.speculative.execution</name>71 <value>false</value>72 </property>73 <property>74 <name>mapred.reduce.tasks.speculative.execution</name>75 <value>false</value>76 </property>77 <property>78 <name>mapred.reduce.slowstart.completed.maps</name>79 <value>0.8</value>80 </property>81 </configuration>
Saturday, August 24, 13
![Page 43: Growing Pains: Avoiding Module Hell](https://reader034.vdocuments.site/reader034/viewer/2022042518/5595a1801a28ab00448b4768/html5/thumbnails/43.jpg)
{ "foobar-cluster": { "mapred-options": [ { "name": "mapred.map.output.compression.codec", "value": "org.apache.hadoop.io.compress.SnappyCodec" }, { "name": "mapred.compress.map.output", "value": "true" }, { "name": "io.sort.factor", "value": "64" }, { "name": "io.sort.record.percent", "value": "0.05" }, { "name": "mapred.map.tasks.speculative.execution", { "name": "mapred.reduce.tasks.speculative.execution",54 "value": "false"55 },56 {57 "name": "mapred.reduce.slowstart.completed.maps",58 "value": "0.8"59 }60 ]61 }62 }
Saturday, August 24, 13
![Page 44: Growing Pains: Avoiding Module Hell](https://reader034.vdocuments.site/reader034/viewer/2022042518/5595a1801a28ab00448b4768/html5/thumbnails/44.jpg)
Saturday, August 24, 13