we open sourced our trade secrets, and so can you!
DESCRIPTION
AppNeta TraceView is distributed tracing software as a service. Our Ruby instrumentation is a gem that hooks into popular frameworks, clients, and drivers to capture performance data from a running application – and if you think storing data in the cloud is a hard sell, just wait until you mention that you're modifying executing code to capture it! Bad instrumentation could have all kinds of adverse impacts on your application, from crashes to slowdowns to subtle corruptions, so our guiding principle is "first, do no harm". Plenty of customers have found our gem to be fast and stable under real-world conditions, but until recently, they had to take us at our word that we weren't shipping backdoors, ticking time bombs, or dark magic. About a year ago, we decided that it was time to open source our gem. Starting off proprietary meant that we'd never planned for the world to read our code, so we didn't yet understand what risks we were exposing ourselves and our customers to. It was also internal-focused, so we weren't even sure whether anyone who wasn't an AppNeta engineer would get anything out of it. But we wanted to make it happen, and for it to happen to more code in the future, so we developed an internal process for open-sourcing code sustainably. Before launch day, we got buy-in from involved departments, vetted the license with our lawyers, figured out how our support team would support it, audited code and removed information that needed to stay private, wrote tests that anyone could run, turned internal documentation into "TFM", figured out our internal workflow, and developed a framework for community contributions. It took a few months to get everything in order, but the day finally came when we switched over our RubyGems build to the new repo and opened it to the world. (I wrote the announcement blog post – yay!) Our gem has been on GitHub at appneta/oboe_ruby for over six months now, and we're very pleased with the results we've seen. We've had no problem continuing to sell our product, our Ruby customers have been much more engaged, and we've even been able to release new instrumentation based on their contributed code. We're glad we open sourced our trade secrets – what's stopping you?TRANSCRIPT
We open sourced our trade secrets, and so can you!
James MeickleDeveloper evangelist, AppNeta
@jmeickle
Boston Rails MeetupAugust 26, 2014
AppNeta
• Application performance management (APM)– Client– Network– Server
• Software as a service• Cloud hosted
A brief history of APM
• Closed source monitoring for closed source applications
• Oriented around mainframes and J2EE
• Incredibly high cost, not production-ready
• Today: rapid value, easy deployment, low cost
AppNeta TraceView
• Distributed tracing (Zipkin, Dapper)
• Bytecode instrumentation
• Java, .NET, PHP, Ruby, Python, node.js
• Definitely not Tracelytics!
THE GEMImplementing TraceView’s functionality in Ruby
Our maintainer
• Peter Lombardo came on a few years ago
• Exclusively focused on the oboe gem
• Works remote from Italy, or else he’d be giving this talk!
oboe
• Installed like a normal gem (with C extension)
• No configuration required• Provides:
– Application performance management
– Software as a service– Distributed tracing– Bytecode instrumentation– Trace API
Application performance management
• Collect performance data:– Database queries– Cache requests– Remote calls– Templating time
• Without causing problems:– Identical behavior– Overhead below 1%– Stable
Software as a service
• Don’t do any processing locally
• Report data to the agent (UDP)
• Report data to collector (SSL, on Heroku)
Distributed tracing
• Follow calls from one application to another– Different servers– Multiple languages– Asynchronous
• Common request/response protocols:– HTTP– Thrift– EJB (for JRuby)
Bytecode instrumentation
• Support common components:– Frameworks– Libraries– Drivers/clients
• Without code modification– Ruby, Python: monkey
patching– Java: classloader– PHP:
Trace API
• Same API that we use when adding layers
• Our backend is built around supporting it
• Duck typing for events• Report anything!*
*please don’t report credit card numbers, thanks
Example 1: memcache
Example 2: rack
Example 2: Net::HTTP
Example 4: resque
THE PROCESSOpening the oboe gem to the world
Why did we open source oboe?
• We <3 open source• Larger customers want
to audit our code• Increased community
contributions• Better capture of
reported issues• Good publicity
Why couldn’t we “flip the switch”?
• The code was not appropriately documented
• The test suite needed work• There was no contribution
guide• It wasn’t licensed• Not branded properly!• Issues contained
customer-specific information
Documentation
• Added
Tests• “Before opensourcing, we had to
move away from manual testing via nosetests (30 something stack variations, manual boot and manually run nosetests, rinse and repeat for the next hour).
We went with Minitest. Today with Minitest, we now test 7 Ruby versions: each with 202 tests, 1979 assertions - biggest win beyond coverage is that it’s automated with Travis for each git commit.”
– Peter Lombardo
Contribution guide
Branding
• We were already AppNeta TraceView
• But we still were Tracelytics in our code
• Not as simple as a s/foo/bar/; lots of unexpected places (support URLs!)
Licensing
• Open source != free software
• Competitive advantage• Still an unfathomable
change from where we were 10 years ago
Issues with issues
• We didn’t want to lose historical issues
• New issues would also need to reference specific customers and post about their data
The solution
• “The key was to use two repos (1 private and 1 public) using two git remotes. Now our default is that all issues are filed in the public repository unless it contains customer specific data. If a public issue, turns out to need a pointer to some internal data or resource, we file a corresponding internal issue that points to the public issue. We now always prefer the public repo over the private and only use the private when absolutely necessary.”
– Peter Lombardo
The final countdown
Marketing helped too!
• Pre-launch– Implications of licensing– How to reward and
publicize contributors– Info page for repo
• Post-launch– Launch blog post– Social media– Existing customers!
Launch day
• Set up the public repo as a private one
• Switch our build process (rubygems) to use the right repo
• Post the new build as a GitHub Release too
• Publish the blog post!
THE RESULTWhat we’ve seen after open sourcing
appneta / oboe-ruby
• Not many visitors • But they were incredibly
engaged: more than 25% of them cloned the repo.
• And about a third of them contributed code!
More contributions
• 80%: post an issue• 15%: post a fix/snippet• 5%: an entire feature!
– Grape API: Todd Lunter at Swipely
– EventMachine: Diogo Benica at Abril Midia
Happier maintainer• “Not sure if this part is going to make
sense but working on an opensource project is like having a weight lifted off of your shoulders. When it’s private, proprietary code, the burden is completely on you to assure quality and to not screw up. When the code is opensource, there is a slightly calming feel because you know that your code is public and is available to be reviewed and improved upon by anyone. We love the contributions we’ve gotten so far and are always looking for more.”
– Peter Lombardo
GitHub issues
• Work being done lives in GitHub issues on the public repo
• Anyone can post there!
Supporting contributors
• Most people use the trace API incorrectly
• We only notice when they file support tickets
• Post custom code on GH, and we can help while you write it!
Thank you!Come work for us! We’re a mostly Python shop
You can meet us at:
• Sep 15-17: Velocity New York• Sep 18: WebPerfDays New York• Oct 21: TechBreakfast New York• October 24: Surge (DC)• Nov 11-14: AWS re:Invent
(Vegas!)
Or just try us out:
http://www.appneta.com/products/traceview/