your tests are not your specs
TRANSCRIPT
Tests are not specsHow to write actual specifications with TLA+
Hillel Wayne@hillelogram
Rules• Interrupt me if I’m speaking too fast• You don’t need to understand the code samples• Ask questions if you have them• I may punt them until later
Espark LearningOr core, same thing
MDM
School
MDM
School
Business logic
TDD• Write a failing test• Fix the failing test• Refactor
Describe EmailSender context “when called” it “should send an email” it “should not send two emails”
Describe EmailSender context “when called” context “and the email API is lagging out” it “should send an email” it “should not send two emails”
Describe EmailSender context “when called” context “and the email API is lagging out” context “and there are three separate email services” context “and our AP database has a partition” it “should send an email” it “should not send two emails”
TESTS ARE NOT SPECS
Programming Languages
• I/O• Implementations• Concrete• Fast• Designed to run
Specification Languages
• No I/O• Definitions• Abstract• “Comprehensive”• Designed to check
Maximum of Setdef maximum(numbers) max = numbers.first numbers.each do |n| if n > max max = n end maxend
Maximum(numbers) == CHOOSE max \in numbers: \A n \in numbers: max >= n
Alice BobMoney
Variables alice_account = 4, bob_account = 0;
Process transfer \in {1}Variables to_transfer \in 1..5Begin Withdraw: alice_account := alice_account – to_transfer; Deposit: bob_account := bob_account + to_transfer;End process;
NoOverDrafts == alice_account >= 0
1 Withdraw Deposit
2 Withdraw Deposit
3 Withdraw Deposit
4 Withdraw Deposit
5 Withdraw Broken
Variables alice_account = 4, bob_account = 0;
Process transfer \in {1}Variables to_transfer \in 1..5Begin Start: if to_transfer <= alice_account then Withdraw: alice_account := alice_account – to_transfer; Deposit: bob_account := bob_account + to_transfer; end if;End process;
Variables alice_account = 4, bob_account = 0;
Process transfer \in {1, 2}Variables to_transfer \in 1..5Begin Start: if to_transfer <= alice_account then Withdraw: alice_account := alice_account – to_transfer; Deposit: bob_account := bob_account + to_transfer; end if;End process;
CheckWithdraw
CheckWithdraw
Deposit
WithdrawWithdrawCheck
Variables alice_account = 4, bob_account = 0;
Process transfer \in {1}Variables to_transfer \in 1..5Begin Start: if to_transfer <= alice_account then Withdraw: alice_account := alice_account – to_transfer; Deposit: bob_account := bob_account + to_transfer; end if;End process;
<>[](alice_account + bob_account = 4)
Check Withdraw Crash
Use case
• For performance reasons, we should install the app on more than one device per API call.• For stability reasons, we could never show the MDM it was incorrect
about something. So if we tell it to install an app it thinks it installed, the system wigs out.
EXTENDS Integers, TLC, SequencesCONSTANTS Devices
(* --algorithm BatchInstallvariables AppScope \in [Devices -> {0, 1}]; Installs \in [Devices -> BOOLEAN]; batch_pool = {}; lock = FALSE;
define PoolNotEmpty == batch_pool # {}end define
procedure ChangeAppScope()variables cache;begin GetLock: await ~lock; lock := TRUE; Cache: cache := batch_pool; Filter: cache := cache \intersect {d \in Devices: AppScope[d] = 0}; Add: AppScope := [d \in Devices |-> IF d \in cache THEN AppScope[d] + 1 ELSE AppScope[d] ]; Clean:
batch_pool := batch_pool \ cache; lock := FALSE; return;end procedure
fair process SyncDevice \in Devicesbegin Sync: if Installs[self] then batch_pool := batch_pool \union {self}; end if; if PoolNotEmpty then either call ChangeAppScope(); or skip; end either; end if;end process;
fair process TimeLoop = 0begin Start: while TRUE do await PoolNotEmpty; call ChangeAppScope(); end while;end processend algorithm;
Filter: cache := cache \intersect {d \in Devices: AppScope[d] = 0};Clean: batch_pool := batch_pool \ cache;
GetLock: await ~lock; lock := TRUE;
TLA+ SUCKS
Slow
Unfriendly• F[a] vs F(a) vs F a• [A -> B] vs [A |-> B]• = vs == vs := vs /= vs #• Whitespace is sometimes significant• …And More!
SPECS ARE NOT TESTS
Conclusions• Tests check implementation of code• Specs check design of system• Formal specification helps find complicated bugs• TLA+ will save your butt
Resources• http://lamport.azurewebsites.net/tla/tla.html• http://lamport.azurewebsites.net/tla/book.html• https://learntla.com (me!)
Questions?
Hillel Wayne
Twitter: @hillelogram
Site: www.learntla.com