An introduction and future of Ruby coverage library
RubyKaigi 2017 (19th Sep. )
Yusuke Endoh (@mametter)
Yusuke Endoh (@mametter)
• Ruby committer (2008—)
• Full-time Ruby committer (2017—)
• My goal: make Ruby programs robust
– Test coverage, and type system?
Method.Put the cream cheese into the mixing bowl.Put the sour cream into the mixing bowl.Put the white sugar into the mixing bowl.Put the yogrut into the mixing bowl.Put the unsalted butter into the mixing bowl.Combine the milk.Put the cake flour into the mixing bowl.Combine the corn starch.Put the brown sugar into the mixing bowl.Combine the egg whites.Combine the egg yolks.Put the lemon juice into the mixing bowl.Stir for 7 minutes.Liquify the contents of the mixing bowl.Pour contents of the mixing bowl into the baking dish.Bake the cake mixture.Watch the cake mixture until baked.
Serves 4.
Cheese cake in Chef.
Ingredients.100 g cream cheese97 g sour cream107 g yogrut112 g white sugar11 g brown sugar37 g unsalted butter37 g cake flour3 g corn starch3 ml milk3 egg yolks3 egg whites10 ml lemon juice0 g cake mixture
Cooking time: 80 minutes.
‘d’‘a’
‘k’‘p’
11*3*3=99 ‘c’
37*3=111 ‘o’
¥n
‘o’
Push characters‘¥n’ ‘d’ ‘a’ ‘p’ ‘k’ ‘o’ ‘o’ ‘c’
into the stack
Convert them to a stringand “serve” it.
data
code
Esoteric Recipe
• Polyglot of Chef andreal recipe
– Japanese versionhttps://cookpad.com/recipe/4649810
– English versionhttps://cookpad.com/us/recipes/3335222
My main contributions for Ruby
• Implementation of some features:keyword arguments, deadlock detection, etc.
• Release management for Ruby 1.9.2 and 2.0.0
• Optcarrot: A NES emulator for Ruby3x3 benchmark
• Enhancement ofthe test suite of Ruby
• coverage.so: the core libraryfor coverage measurement
’06B ’07A ’07B ’08A60
70
80
90
100
covera
ge (
%)
70%
85%
line coverage
Today’s theme
• An introduction of test coverage
• An improvement plan of Ruby’s coverage measurement feature towards 2.5
Survey [1/3]
• Q. Do you use Ruby/RoR in production?
– Raise your hand, please!
Survey [2/3]
• Q. In those, do you test your code?
Survey [3/3]
• Q. In those, do you use “coverage”?
– Do you check the result of SimpleCov?
Agenda
• What is coverage
• How to understand and use coverage
• The current status of Ruby coverage feature
• The future plan of Ruby coverage feature
• Conclusion
Agenda
☞ What is coverage
• How to understand and use coverage
• The current status of Ruby coverage feature
• The future plan of Ruby coverage feature
• Conclusion
What is coverage?
• A measure of “goodness” of a test suite– Also called “test coverage” or “code coverage”
• Allows you:– Find untested code– Decide whether your test suite is good enough
or not yet• (This is arguable, but I think we can use it as an
advice)
• Types of coverage– Function coverage, line coverage, branch
coverage, …
Function coverage
• How many functions are executed by the tests# codedef foo; …; end # ✓def bar; …; end # ✗def baz; …; end # ✓
# testfoobaz
2/3(67%)
• Advantage• Easy to understand
• Easy to visualize
• Disadvantage• Too weak as a measure
Line coverage
• How many lines are executed
# codedef foo(x) # ✓
if x == 0 # ✓p :foo # ✗
elsep :bar # ✓
endend
# testfoo(1)
3/4(75%)
Non-significant line isignored
• Advantage• Easy to understand• Easy to visualize
• Disadvantage• Still weak as a measure
• foo() if x == 0
Branch coverage
• How many branches are taken true and false
# codedef foo(x)
p :foo if x == 0 # ✓p :bar if x < 2 # ✗
end
# testfoo(0)foo(1)
1/2(50%)
• Advantage• Relatively exhaustive
• Disadvantage• Difficult to visualize
true-case and false-case areboth executed
Coverage types
Coverage type Easy tounderstand/
visualize
Exhaustive
Functioncoverage
○ ✕
Linecoverage
○ △
Branchcoverage
△ ○
Currently, Ruby supports only line coverage
Other types of coverage
• Condition coverage– How many conditions
(not branches) are takenboth true and false
• Path coverage– How many paths are executed
– Combinatorial explosion
• Other advanced ones– Data-flow coverage
– MC/DC coverage
if a && b
branch
conditioncondition
Trivia
• “C0/C1/C2 coverages” have difference meanings to different people
– C0 coverage = line coverage
– C1 coverage = branch coverage or path coverage?
– C2 coverage = condition coverage or path coverage?
Coverage and Ruby
• In Ruby, Coverage is crucial!– A test is the only way to ensure quality– Coverage is important to measure test goodness
• Considering it, coverage is not used so much…– Coverage is not well-known?– It is not well-known how to use coverage?– Ruby’s coverage measurement feature is not
enough?
• I’d like to improve the situation with this talk
Agenda
• What is coverage
☞ How to understand and use coverage
• The current status of Ruby coverage feature
• The future plan of Ruby coverage feature
• Conclusion
What is a good test suite?
• Covers the code
– Coverage measures this
• Also covers the design of program
– Coverage does not measure this directly
How to understand coverage
• Coverage is just a measure
• Coverage is not a goal
If coverage XX% is required as a goal…
• Developers will1. Pick untested code that looks easiest to
cover
2. Write a trivial test just for the code
3. Iterate this procedure until XX% is achieved
• It will result in trivial, not-so-good test suite– It may be exhaustive for the code itself, but
– It won't be exhaustive for the design
A good way to improve coverage
• Developers should1. Look at untested code
2. Consider what “test design” is insufficient
3. Write them
– In consequence of them, the untested code is executed
• It will result in good test suite– It will be exhaustive not only for the code
but also for the design
How many % is needed/enough?
• It depends upon the module being tested– Aim 100% for a significant module (e.g., injure
someone)
– Don't worry too much for a non-significant module
• It also depends upon cost performance– For non-significant module, write a test only if it
is not so hard
• Again: Coverage is not a goal
Agenda
• What is coverage
• How to understand and use coverage
☞ The current status of Ruby coverage feature
• The future plan of Ruby coverage feature
• Conclusion
Ruby's coverage ecosystem
• SimpleCov
• coverage.so
• Concov
SimpleCov
• A wrapper library for coverage.so• Visualization with HTML• Useful features: merging, filtering, for Rails app• Author: Christoph Olszowka (@colszowka)
Usage of SimpleCov
• Write this at the top of test/test_helper.rb
• Run the test suite
• coverage/index.html will be produced
– Note: SimpleCov cannot measure already-loaded files before SimpleCov.start
require "simplecov"SimpleCov.start
coverage.so
• The only implementation of coverage measurement for Ruby 1.9+
• SimpleCov is a wrapper for coverage.so
• Author: me
Basic usage
# test.rbrequire "coverage"Coverage.start
load "target.rb"
p Coverage.result#=> {"target.rb"=># [nil,nil,1,1,1,nil,# 0,nil,nil,nil,1]}
Start measuring coverage
Load the target file
Stop measuringand get the result
Coverage data
Coverage data
# target.rb
def foo(x)if x == 0
p 0else
p 1end
end
foo(1)
[nil,nil110
nil1
nilnilnil1]
nil:Non-significant line
Number:Count executed
Untested line!
Method definition is code in Ruby
# target.rb
def foo(x)if x == 0
p 0else
p 1end
end
[nil,nil100
nil0
nilnil]
Method definition iscounted as an
execution
(It is not a count ofmethod invocation!)
I regret the design of coverage.so
• Support only line coverage• Excuse: I introduced it just for test
enhancement of Ruby itself– I didn't deliberate the API for external
project…
• I have wanted to make the next version better
ext/coverage/coverage.c:
69 /* Coverage provides coverage measurement feature for Ruby.70 * This feature is experimental, so these APIs may be changed in future.71 *
Concov
• CONtinuous COVerage– Detects temporal change (degradation) of
coverage
• Developed for monitoring Ruby's coverage
• Author: me
• Presented at RubyKaigi 2009, and then…– It has not been used by everyone (including me)
– It was based on Ramaze (old web framework)!
Concov reveals reality of Ruby dev.(Enumerable#join, 2009/07/07, M***)
New feature
introduced
with no tests!
Concov reveals reality of Ruby dev.(File#size, 2009/02/25, M***)
Concov reveals reality of Ruby dev.(Etc::Passwd.each, 2009/02/19, N*****)
Concov reveals reality of Ruby dev.(Dir.home, 2009/02/03, N*****)
Concov reveals reality of Ruby dev.(Array#sort_by!, 2009/02/03, M***)
Concov reveals reality of Ruby dev.(rb_to_float, 2008/12/30, M***)
Coverage ecosystem for other languages
• C/C++: GCOV/LCOV– Integrated with gcc:gcc -coverage target.c
• Java: A lot of tools– Cobertura, Emma,
Clover, JaCoCo– Integrated with CI tools
and/or IDEs
• JavaScript: IstanbulJenkins Cobertura plugin
LCOV result
Agenda
• What is coverage
• How to understand and use coverage
• The current status of Ruby coverage feature
☞ The future plan of Ruby coverage feature
• Conclusion
A plan towards Ruby 2.5
• Support function and branch coverage
– There have been multiple requests and some PoC patches…
• To make the API better, any comments are welcome
– https://bugs.ruby-lang.org/issues/13901
API: to start measuring
# compatible layerCoverage.startCoverage.result#=> {"file.rb"=>[nil, 1, 0, …], … }
# new APICoverage.start(lines: true)Coverage.result#=> {"file.rb" => { :lines => [nil, 1, 0, …] } }
API: to start other types of coverage
# enable branch and function coverageCoverage.start(lines:true,
branches:true,methods:true)
Coverage.result#=> {"file.rb" => { :lines => [nil, 1, 0, …],# :branches => {…},# :methods => {…} } }
# shorthandCoverage.start(:all)
Coverage.result for branch coverage
{"target1.rb"=>{:lines=>[…],
:branches=>{[:if, 0, 2]=>{
[:then, 1, 3]=>2,[:else, 2, 5]=>1
}},:methods=>{
[:test_if, 1]=>3}}}
# target1.rb1: def test_if(x)2: if x == 03: p "x == 0"4: else5: p "x != 0"6: end7: end8: 9: test_if(0)10: test_if(0)11: test_if(1)
From if at Line 2
Jumped tothen clauseat Line 3
twice
Jumped toelse clauseat Line 5
once
Coverage.result for branch coverage
{"target2.rb"=>{:lines=>[1, 1, 1, nil, nil, 1],
:branches=>{[:if, 0, 2]=>{
[:then, 1, 2]=>1,[:else, 2, 2]=>0},
[:if, 3, 3]=>{[:then, 4, 3]=>0,[:else, 5, 3]=>1}},
:methods=>{[:test_if, 1]=>3
}}}
# target2.rb1: def test_if_oneline(x)2: p "x == 0" if x == 03: p "x != 0" if x != 04: end5: 6: test_if_oneline(0)
Line coverage 100%
Branch coverage tells youthere are untested cases
Discussion of API design
• 100% compatible
• [<label>, <numbering>, <line-no>]– e.g., [:if, 0, 1], [:while, 1, 1], [:case, 2, 1]– <numbering> is a unique ID to avoid conflicts for the
case where there are multiple branches in one line• LCOV-style
– Other candidates:• [<label>, <line-no>, <column-no>]
– How to handle TAB character
• [<label>, <offset from file head>]– Looks good, but hard to implement (I'll try later)
Overhead of coverage measurement
(just preliminary experiment)
# Example 21: foo()2: foo()…
99: foo()100: foo()
Benchmark w/o cov. w/ cov. Overhead
Example 1 0.322 μs 6.21 μs x19.3
Example 2 1.55 μs 7.16 μs x4.61
make test-all 485 s 550 s x1.13
# Example 11: x = 12: x = 1…99: x = 1100: x = 1
Demo
• Applied the new coverage.so to Ruby
• Integrated with C code coverage by GCOV and Visualized by LCOV
Ruby codein stdlib
make exam withgcc -coverage
make examCOVERAGE=1
test-coverage
.dat
*.gcda gcov
my script
run test
C codeof MRI
cov. datasource aggregate
lcov HTML
Jenkins Cobertura Plugin
Agenda
• What is coverage
• How to understand and use coverage
• The current status of Ruby coverage feature
• The future plan of Ruby coverage feature
☞ Conclusion
Acknowledgement
• @_ko1• @t_wada• @kazu_cocoa• @moro• @makimoto• @dev_shia• @tanaka_akr• @nalsh• @spikeolaf• @k_tsj
Conclusion
• What is coverage, how important in Ruby, and how to understand coverage
• The current status of Ruby's coverage measurement and ecosystem
• A plan towards Ruby 2.5 and preliminary demo– Any comments are welcome!
– https://bugs.ruby-lang.org/issues/13901
Future work
• Determine the API
• define_method as method coverage
• &. as branch coverage
• Callsite coverage
• Block coverage
obj.foo.bar
ary.map { …… }