How to Write Better Code with Mutation Testing

John Backus - @backus - john@blockscore.com

Talk Outline

Introduction

Talk Outline

Introduction

Improving coverage

Talk Outline

Introduction

Improving coverage

Learning about Ruby and the code you rely on

Line Coverage

Lines of Code Run by Tests Total Lines of Code÷

Mutation Coverage

How much of your code can I change

without breaking your tests?

Mutation Testing by Hand

1 class Gluttons 2 def initialize(twitter_client) 3 @twitter = twitter_client 4 end 5 6 def recent 7 query = @twitter.search('"I really enjoy #pizza"') 8 9 query.first(2).map { |tweet| "@#{tweet.author}" } 10 end 11 end

1 RSpec.describe Gluttons do 2 it 'lists the two most recent gluttonous tweeters' do 3 tweets = [double(author: 'John'), double(author: 'Jane')] 4 gluttons = Gluttons.new(double(search: tweets)) 5 6 expect(gluttons.recent).to eql(%w[@John @Jane]) 7 end 8 end

1 class Gluttons 2 def recent 3 - query = @twitter.search('"I really enjoy #pizza"') 4 + query = @twitter.search('"I really enjoy #hotdogs"')

1 example, 0 failures

1 class Gluttons 2 def recent 3 - query = @twitter.search('"I really enjoy #pizza"') 4 + query = @twitter.search('')

1 example, 0 failures

1 class Gluttons 2 def recent 3 - query = @twitter.search('"I really enjoy #pizza"') 4 + query = @twitter.search

1 example, 0 failures

1 class Gluttons 2 def recent 3 - query.first(2).map { |tweet| "@#{tweet.author}" } 4 + query.first(1).map { |tweet| "@#{tweet.author}" }

Failure

expected: ["@John", "@Jane"] got: ["@John"]

1 - query.first(2).map { |tweet| "@#{tweet.author}" } 2 + query.first(3).map { |tweet| "@#{tweet.author}" } 3 end

1 example, 0 failures

Manual Mutation Testing

Tedious

Manual Mutation Testing

Hard to outsmart yourself

1 def recent 2 - query = @twitter.search('"I really enjoy #pizza"') 3 + query = @twitter.search

1 def recent 2 - query = @twitter.search('"I really enjoy #pizza"') 3 + query = @twitter.search(nil)

1 - query.first(2).map { |tweet| "@#{tweet.author}" } 2 + query.last(2).map { |tweet| "@#{tweet.author}" } 3 end

1 - query.first(2).map { |tweet| "@#{tweet.author}" } 2 + query.map { |tweet| "@#{tweet.author}" } 3 end

1 it 'lists the two most recent gluttonous tweeters' do 2 tweets = [ 3 double(author: 'John'), 4 double(author: 'Jane'), 5 double(author: 'Devon') 6 ] 7 8 client = double('Client') 9 gluttons = Gluttons.new(client) 10 11 allow(client) 12 .to receive(:search) 13 .with('"I really enjoy #pizza"') 14 .and_return(tweets) 15 16 expect(gluttons.recent).to eql(%w[@John @Jane]) 17 end

Mutation Testing with mutant

Automated

Probably more clever than you

Mutation Testing with mutant

Example #1: Internal API

1 it 'returns a user when given a valid id' do 2 expect(get(:show, id: 1)).to eq(id: 1, name: 'John') 3 end 4 5 it 'renders JSON error when given an invalid id' do 6 expect(get(:show, id: 0)) 7 .to eq(error: "Could not find User with 'id'=0") 8 end

1 class UsersController < ApplicationController 2 def show 3 render json: User.find(params[:id].to_i) 4 rescue User::RecordNotFound => error 5 render json: { error: error.to_s } 6 end 7 end

1 def show 2 - render json: User.find(params[:id].to_i) 3 + render json: User.find(Integer(params[:id])) 4 rescue User::RecordNotFound => error 5 render json: { error: error.to_s } 6 end

1 def show 2 - render json: User.find(params[:id].to_i) 3 + render json: User.find(params.fetch(:id).to_i) 4 rescue User::RecordNotFound => error 5 render json: { error: error.to_s } 6 end

1 def show 2 - render json: User.find(params[:id].to_i) 3 + render json: User.find(Integer(params.fetch(:id))) 4 rescue User::RecordNotFound => error 5 render json: { error: error.to_s } 6 end

1 class UsersController < ApplicationController 2 def created_after 3 after = Date.parse(params[:after]) 4 render json: User.recent(after) 5 end 6 end

1 def created_after 2 - after = Date.parse(params[:after]) 3 + after = Date.iso8601(params[:after]) 4 render json: User.recent(after) 5 end

1 def created_after 2 - after = Date.parse(params[:after]) 3 + after = Date.iso8601(params[:after]) 4 render json: User.recent(after) 5 end

1 def created_after 2 - after = Date.parse(params[:after]) 3 + after = Date.iso8601(params[:after]) 4 render json: User.recent(after) 5 end

“2017-05-01""H29.05.01""Tue May 01 00:00:00 2017""Tue, 01 May 2017 00:00:00 +0000""Tue, 01 May 2017 00:00:00 GMT""May""I may be complete garbage"

“2017-05-01"

Date.parseDate.iso8601

Example #2: Hardening Regular Expressions

1 usernames.select do |username| 2 username =~ /^(John|Alain).+$/ 3 end

1 usernames.select do |username| 2 - username =~ /^(John|Alain).+$/ 3 + username =~ /\A(John|Alain).+$/ 4 end

1 usernames.select do |username| 2 - username =~ /^(John|Alain).+$/ 3 + username =~ /^(John|Alain).+\z/ 4 end

1 usernames.select do |username| 2 - username =~ /^(John|Alain).+$/ 3 + username =~ /^(Alain).+$/ 4 end

1 usernames.select do |username| 2 - username =~ /^(John|Alain).+$/ 3 + username =~ /^(John).+$/ 4 end

1 usernames.select do |username| 2 - username =~ /^(John|Alain).+$/ 3 + username =~ /^(?:John|Alain).+$/ 4 end

1 usernames.select do |username| 2 - username =~ /^(John|Alain).+$/ 3 + username.match?(/^(John|Alain).+$/) 4 end

1 usernames.select do |username| 2 - username =~ /^(John|Alain).+$/ 3 + username.match?(/\A(?:John|Alain).+\z/) 4 end

Example #3: Learning about your HTTP Client

1 def stars_for(repo) 2 url = "https://api.github.com/repos/#{repo}" 3 data = HTTParty.get(url).to_h 4 5 data['stargazers_count'] 6 end

1 def stars_for(repo) 2 url = "https://api.github.com/repos/#{repo}" 3 - data = HTTParty.get(url).to_h 4 + data = HTTParty.get(url) 5 6 data['stargazers_count'] 7 end

Practicality

/^(John|Alain).+$/

/\A(John|Alain).+$/

/^(John|Alain).+\z/

/^(Alain).+$/

/^(John).+$/

/^(?:John|Alain).+$/

$ mutant --use rspec --since master

$ mutant \--use rspec \--since master \‘YourApp::User’

1 module YourApp 2 class User < ActiveRecord::Base 3 # Dozens of includes, scopes, class methods, rails magic 4 # 100+ methods 5 6 def validate_email 7 # Simple method you're fixing 8 end 9 end 10 end

1 RSpec.describe YourApp::User do 2 # 100s of tests and setup unrelated to your task 3 4 describe '#validate_email' do 5 # Half dozen tests you are focusing on 6 end 7 end

mutant - Your Secret Weapon

Thanks!

John Backus - @backus - john@blockscore.com