EECS 491Introduction to Distributed

Systems

Fall 2019

Harsha V. Madhyastha

Primary Backup Replication

September 17, 2019 EECS 491 – Lecture 5 2

Client Primary Backup

Backup

Backup

Primary Backup Replication● Promote one of the backups if primary fails● Replace any failed backup

● When should primary sync with backups?◆ Before making state change externally visible◆ Primary and backups must be externally consistent

● What to sync?◆ Entire state when bootstrapping new backup◆ Thereafter, forward every update

September 17, 2019 EECS 491 – Lecture 5 3

Primary Backup Replication

Client Primary Backup

Backup

Backup

View service

September 17, 2019 EECS 491 – Lecture 5 4

View service● Maintains current membership of primary-

backup service (called view)◆ View number, primary, backup

● When does view service change view?● When primary or any backup fails● Periodically exchange heartbeat messages to

detect failures

September 17, 2019 EECS 491 – Lecture 5 5

Transitioning between views● Why are backups included in current view?● Clients query view service only for primary

◆ Can only promote a previous backup as new primary

● How does view service know if backup up-to-date?● Two scenarios for ill-timed primary failure:

◆ Primary applies operation but fails before syncing with backup

◆ Primary fails before new backup is initialized

September 17, 2019 EECS 491 – Lecture 5 6

Transitioning between views● View change has three steps:

◆ View service announces new view◆ Primary syncs with new backup if there is one◆ Primary acknowledges new view

● View service knows backup is up-to-date once it receives ACK for new view from primary

● Stuck if primary fails in midst of view change● Liveness detection timeout > State transfer time

September 17, 2019 EECS 491 – Lecture 5 7

Scalability of View service● Too much load on view service if all clients

contacts it before every operation

● Clients can cache view across operations

● When to invalidate cached view?● When no/negative response from primary

September 17, 2019 EECS 491 – Lecture 5 8

Split Brain

Client

S1

S2

View service(1,S1, _)

September 17, 2019 EECS 491 – Lecture 5 9

Construct scenario whereprimary that client contactsis not the primary, but itthinks it is

Split Brain

Client

S1

S2

View service(1,S1, _)(2,S1,S2)(3,S2, _)

(1,S1, _)(2,S1,S2)

(2,S1,S2)(3,S2, _)

(2,S1,S2)

September 17, 2019 EECS 491 – Lecture 5 10

Avoiding Split Brain

● Primary must forward all operations to backups◆ Goal: Get ACKs from backups that they too

recognize primary

● Why can’t backups be mistaken about who is primary?◆ Only a backup can be promoted as primary

September 17, 2019 EECS 491 – Lecture 5 11

View service

● Valid sequence of views:◆ (1, S1, _) à (2, S1, S2) à (3, S1, S3) à (4, S3, S4) à (5, S4, _)

● Examples of invalid transitions between views?◆ (1, S1, S2) à (2, S3, S4)◆ (1, S1, S2) à (2, _, S2)◆ (1, S1, _) à (2, S2, S1)

September 17, 2019 EECS 491 – Lecture 5 12

Announcements● Project 1 due on Thursday● Make sure to test on CAEN

◆ Ensure changes only to *impl.go files

● How to detect failed RPCs?◆ Only rely on return value◆ Contents of reply argument set only upon success◆ Contents of reply undefined for failed RPCs

September 17, 2019 EECS 491 – Lecture 5 13

Service Development

● Getting coordination right between primary and backups is tricky◆ Easy to mess up

● Must make replication transparent to developer

September 17, 2019 EECS 491 – Lecture 5 14

Transparent Replication

15

Replicated State Machine

Application

Updates

Ordered Updates

September 17, 2019 EECS 491 – Lecture 5

Replicated State Machine

Application

Updates

Server1 Server2

Transparent Primary Backup

● Application relies on library to keep primary and backups in sync◆ Receive message from client◆ Sync with backups before sending response to client

● Will this solution work?

September 17, 2019 EECS 491 – Lecture 5 16

Example: Bank Serverlast_month uint

Deposit(user, amount) {

balance[user] += amount

if (curr_month > last_month) {

add 1% monthly interest

last_month = curr_month

}return balance[user]

}September 17, 2019 EECS 491 – Lecture 5 17

Example ExecutionPrimary

● Receive deposit request● Update balance● Read current time T1● Add interest● Return new balance

Backup● Receive deposit request● Update balance● Read current time T2● Return new balance

September 17, 2019 EECS 491 – Lecture 5 18

Insufficient to relay inputs from primary to backupAlso need to relay sources of non-determinism

How to capture this?

Virtual Machines

Operating System

Hardware

Applications

CPU Disk RAM

Process File system Virtual memory

VirtualMachine

Virtual Machine Monitor

September 17, 2019 EECS 491 – Lecture 5 19

RSM with Primary Backup

20

Operating System

Application

September 17, 2019 EECS 491 – Lecture 5

Operating System

Application

Server1 Server2

Virtual Machine Monitor

Virtual Machine Monitor

Hardware Hardware

September 17, 2019 EECS 491 – Lecture 5 21

Primary VM

Backup VM

Logging channel

Shared Disk

�!�,)� �� �*!� �� �&%��,)�+!&%�

@8AG4G<BA B9 94H?GGB?8E4AG .%F 9BE G;8 ( (�*!+� C?4G9BE@� 'HE 4CCEB46; <F F<@<?4E 5HG J8 ;4I8 @478 FB@89HA74@8AG4? 6;4A:8F 9BE C8E9BE@4A68 E84FBAF 4A7 <AI8FG<:4G87 4 AH@58E B9 78F<:A 4?G8EA4G<I8F� !A 477<G<BA J8 ;4I8;47 GB 78F<:A 4A7 <@C?8@8AG @4AL 477<G<BA4? 6B@CBA8AGF<A G;8 FLFG8@ 4A7 784? J<G; 4 AH@58E B9 CE46G<64? <FFH8FGB 5H<?7 4 6B@C?8G8 FLFG8@ G;4G <F 8�6<8AG 4A7 HF45?8 5L6HFGB@8EF EHAA<A: 8AG8ECE<F8 4CC?<64G<BAF� +<@<?4E GB @BFGBG;8E CE46G<64? FLFG8@F 7<F6HFF87 J8 BA?L 4GG8@CG GB 784?J<G; 94<?FGBC 94<?HE8F 1��3 J;<6; 4E8 F8EI8E 94<?HE8F G;4G 64A58 78G86G87 589BE8 G;8 94<?<A: F8EI8E 64HF8F 4A <A6BEE86G 8KG8EA4??L I<F<5?8 46G<BA�,;8 E8FG B9 G;8 C4C8E <F BE:4A<M87 4F 9B??BJF� �<EFG J8

78F6E<58 BHE 54F<6 78F<:A 4A7 78G4<? BHE 9HA74@8AG4? CEBGB6B?F G;4G 8AFHE8 G;4G AB 74G4 <F ?BFG <9 4 546>HC .% G4>8FBI8E 49G8E 4 CE<@4EL .% 94<?F� ,;8A J8 78F6E<58 <A 78G4<? @4AL B9 G;8 CE46G<64? <FFH8F G;4G @HFG 58 477E8FF87 GB5H<?7 4 EB5HFG 6B@C?8G8 4A7 4HGB@4G87 FLFG8@� /8 4?FB78F6E<58 F8I8E4? 78F<:A 6;B<68F G;4G 4E<F8 9BE <@C?8@8AG<A:94H?GGB?8E4AG .%F 4A7 7<F6HFF G;8 GE478B�F <A G;8F8 6;B<68F�&8KG J8 :<I8 C8E9BE@4A68 E8FH?GF 9BE BHE <@C?8@8AG4G<BA9BE FB@8 58A6;@4E>F 4A7 FB@8 E84? 8AG8ECE<F8 4CC?<64G<BAF��<A4??L J8 78F6E<58 E8?4G87 JBE> 4A7 6BA6?H78�

2. BASIC FT DESIGN�<:HE8 � F;BJF G;8 54F<6 F8GHC B9 BHE FLFG8@ 9BE 94H?G

GB?8E4AG .%F� �BE 4 :<I8A .% 9BE J;<6; J8 78F<E8 GB CEBI<7894H?G GB?8E4A68 �G;8 -.'*�.4 .%� J8 EHA 4 �!(1- .% BA4 7<�8E8AG C;LF<64? F8EI8E G;4G <F >8CG <A FLA6 4A7 8K86HG8F<78AG<64??L GB G;8 CE<@4EL I<EGH4? @46;<A8 G;BH:; J<G; 4F@4?? G<@8 ?4:� /8 F4L G;4G G;8 GJB .%F 4E8 <A 2'.01�) ),!(�/0#-� ,;8 I<EGH4? 7<F>F 9BE G;8 .%F 4E8 BA F;4E87 FGBE4:8�FH6; 4F 4 �<5E8 �;4AA8? BE <+�+! 7<F> 4EE4L� 4A7 G;8E89BE8 4668FF<5?8 GB G;8 CE<@4EL 4A7 546>HC .% 9BE <ACHG 4A7BHGCHG� �/8 J<?? 7<F6HFF 4 78F<:A <A J;<6; G;8 CE<@4EL 4A7546>HC .% ;4I8 F8C4E4G8 ABAF;4E87 I<EGH4? 7<F>F <A +86G<BA ����� 'A?L G;8 CE<@4EL .% 47I8EG<F8F <GF CE8F8A68 BAG;8 A8GJBE> FB 4?? A8GJBE> <ACHGF 6B@8 GB G;8 CE<@4EL .%�+<@<?4E?L 4?? BG;8E <ACHGF �FH6; 4F >8L5B4E7 4A7 @BHF8� :BBA?L GB G;8 CE<@4EL .%��?? <ACHG G;4G G;8 CE<@4EL .% E868<I8F <F F8AG GB G;8

546>HC .% I<4 4 A8GJBE> 6BAA86G<BA >ABJA 4F G;8 ),%%'+%!&�++#)� �BE F8EI8E JBE>?B47F G;8 7B@<A4AG <ACHG GE4�6<F A8GJBE> 4A7 7<F>� �77<G<BA4? <A9BE@4G<BA 4F 7<F6HFF8758?BJ <A +86G<BA ��� <F GE4AF@<GG87 4F A868FF4EL GB 8AFHE8G;4G G;8 546>HC .% 8K86HG8F ABA78G8E@<A<FG<6 BC8E4G<BAF<A G;8 F4@8 J4L 4F G;8 CE<@4EL .%� ,;8 E8FH?G <F G;4G G;8546>HC .% 4?J4LF 8K86HG8F <78AG<64??L GB G;8 CE<@4EL .%� BJ8I8E G;8 BHGCHGF B9 G;8 546>HC .% 4E8 7EBCC87 5LG;8 ;LC8EI<FBE FB BA?L G;8 CE<@4EL CEB7H68F 46GH4? BHGCHGFG;4G 4E8 E8GHEA87 GB 6?<8AGF� �F 78F6E<587 <A +86G<BA ��� G;8CE<@4EL 4A7 546>HC .% 9B??BJ 4 FC86<�6 CEBGB6B? <A6?H7<A:8KC?<6<G 46>ABJ?87:@8AGF 5L G;8 546>HC .% <A BE78E GB8AFHE8 G;4G AB 74G4 <F ?BFG <9 G;8 CE<@4EL 94<?F�,B 78G86G <9 4 CE<@4EL BE 546>HC .% ;4F 94<?87 BHE FLF

G8@ HF8F 4 6B@5<A4G<BA B9 ;84EG584G<A: 58GJ88A G;8 E8?8I4AGF8EI8EF 4A7 @BA<GBE<A: B9 G;8 GE4�6 BA G;8 ?B::<A: 6;4AA8?�!A 477<G<BA J8 @HFG 8AFHE8 G;4G BA?L BA8 B9 G;8 CE<@4ELBE 546>HC .% G4>8F BI8E 8K86HG<BA 8I8A <9 G;8E8 <F 4 FC?<G5E4<A F<GH4G<BA J;8E8 G;8 CE<@4EL 4A7 546>HC F8EI8EF ;4I8?BFG 6B@@HA<64G<BA J<G; 846; BG;8E�!A G;8 9B??BJ<A: F86G<BAF J8 CEBI<78 @BE8 78G4<?F BA F8I

8E4? <@CBEG4AG 4E84F� !A +86G<BA ��� J8 :<I8 FB@8 78G4<?FBA G;8 78G8E@<A<FG<6 E8C?4L G86;AB?B:L G;4G 8AFHE8F G;4G CE<@4EL 4A7 546>HC .%F 4E8 >8CG <A FLA6 I<4 G;8 <A9BE@4G<BAF8AG BI8E G;8 ?B::<A: 6;4AA8?� !A +86G<BA ��� J8 78F6E<584 9HA74@8AG4? EH?8 B9 BHE �, CEBGB6B? G;4G 8AFHE8F G;4G AB74G4 <F ?BFG <9 G;8 CE<@4EL 94<?F� !A +86G<BA ��� J8 78F6E<58BHE @8G;B7F 9BE 78G86G<A: 4A7 E8FCBA7<A: GB 4 94<?HE8 <A 46BEE86G 94F;<BA�

2.1 Deterministic Replay Implementation�F J8 ;4I8 @8AG<BA87 E8C?<64G<A: F8EI8E �BE .%� 8K8

6HG<BA 64A 58 @B78?87 4F G;8 E8C?<64G<BA B9 4 78G8E@<A<FG<6 FG4G8 @46;<A8� !9 GJB 78G8E@<A<FG<6 FG4G8 @46;<A8F 4E8FG4EG87 <A G;8 F4@8 <A<G<4? FG4G8 4A7 CEBI<787 G;8 8K46G F4@8<ACHGF <A G;8 F4@8 BE78E G;8A G;8L J<?? :B G;EBH:; G;8 F4@8F8DH8A68F B9 FG4G8F 4A7 CEB7H68 G;8 F4@8 BHGCHGF� � I<EGH4? @46;<A8 ;4F 4 5EB47 F8G B9 <ACHGF <A6?H7<A: <A6B@<A:A8GJBE> C46>8GF 7<F> E847F 4A7 <ACHG 9EB@ G;8 >8L5B4E74A7 @BHF8� &BA78G8E@<A<FG<6 8I8AGF �FH6; 4F I<EGH4? <AG8EEHCGF� 4A7 ABA78G8E@<A<FG<6 BC8E4G<BAF �FH6; 4F E847<A:G;8 6?B6> 6L6?8 6BHAG8E B9 G;8 CEB68FFBE� 4?FB 4�86G G;8 .%�FFG4G8� ,;<F CE8F8AGF G;E88 6;4??8A:8F 9BE E8C?<64G<A: 8K86HG<BA B9 4AL .% EHAA<A: 4AL BC8E4G<A: FLFG8@ 4A7 JBE>?B47���� 6BEE86G?L 64CGHE<A: 4?? G;8 <ACHG 4A7 ABA78G8E@<A<F@A868FF4EL GB 8AFHE8 78G8E@<A<FG<6 8K86HG<BA B9 4 546>HC I<EGH4? @46;<A8 ��� 6BEE86G?L 4CC?L<A: G;8 <ACHGF 4A7 ABA78G8E@<A<F@ GB G;8 546>HC I<EGH4? @46;<A8 4A7 ��� 7B<A:FB <A 4 @4AA8E G;4G 7B8FA�G 78:E478 C8E9BE@4A68� !A 477<G<BA @4AL 6B@C?8K BC8E4G<BAF <A K�� @<6EBCEB68FFBEF ;4I8HA78�A87 ;8A68 ABA78G8E@<A<FG<6 F<78 8�86GF� �4CGHE<A:G;8F8 HA78�A87 F<78 8�86GF 4A7 E8C?4L<A: G;8@ GB CEB7H68G;8 F4@8 FG4G8 CE8F8AGF 4A 477<G<BA4? 6;4??8A:8�.%J4E8 78G8E@<A<FG<6 E8C?4L 1��3 CEBI<78F 8K46G?L G;<F

9HA6G<BA4?<GL 9BE K�� I<EGH4? @46;<A8F BA G;8 .%J4E8 I+C;8E8C?4G9BE@� �8G8E@<A<FG<6 E8C?4L E86BE7F G;8 <ACHGF B9 4 .%4A7 4?? CBFF<5?8 ABA78G8E@<A<F@ 4FFB6<4G87 J<G; G;8 .%8K86HG<BA <A 4 FGE84@ B9 ?B: 8AGE<8F JE<GG8A GB 4 ?B: �?8� ,;8.% 8K86HG<BA @4L 58 8K46G?L E8C?4L87 ?4G8E 5L E847<A: G;8?B: 8AGE<8F 9EB@ G;8 �?8� �BE ABA78G8E@<A<FG<6 BC8E4G<BAFFH�6<8AG <A9BE@4G<BA <F ?B::87 GB 4??BJ G;8 BC8E4G<BA GB 58E8CEB7H687 J<G; G;8 F4@8 FG4G8 6;4A:8 4A7 BHGCHG� �BEABA78G8E@<A<FG<6 8I8AGF FH6; 4F G<@8E BE !' 6B@C?8G<BA <A

31

Primary VM

Backup VM

Logging channel

Shared Disk

�!�,)� �� �*!� �� �&%��,)�+!&%�

@8AG4G<BA B9 94H?GGB?8E4AG .%F 9BE G;8 ( (�*!+� C?4G9BE@� 'HE 4CCEB46; <F F<@<?4E 5HG J8 ;4I8 @478 FB@89HA74@8AG4? 6;4A:8F 9BE C8E9BE@4A68 E84FBAF 4A7 <AI8FG<:4G87 4 AH@58E B9 78F<:A 4?G8EA4G<I8F� !A 477<G<BA J8 ;4I8;47 GB 78F<:A 4A7 <@C?8@8AG @4AL 477<G<BA4? 6B@CBA8AGF<A G;8 FLFG8@ 4A7 784? J<G; 4 AH@58E B9 CE46G<64? <FFH8FGB 5H<?7 4 6B@C?8G8 FLFG8@ G;4G <F 8�6<8AG 4A7 HF45?8 5L6HFGB@8EF EHAA<A: 8AG8ECE<F8 4CC?<64G<BAF� +<@<?4E GB @BFGBG;8E CE46G<64? FLFG8@F 7<F6HFF87 J8 BA?L 4GG8@CG GB 784?J<G; 94<?FGBC 94<?HE8F 1��3 J;<6; 4E8 F8EI8E 94<?HE8F G;4G 64A58 78G86G87 589BE8 G;8 94<?<A: F8EI8E 64HF8F 4A <A6BEE86G 8KG8EA4??L I<F<5?8 46G<BA�,;8 E8FG B9 G;8 C4C8E <F BE:4A<M87 4F 9B??BJF� �<EFG J8

78F6E<58 BHE 54F<6 78F<:A 4A7 78G4<? BHE 9HA74@8AG4? CEBGB6B?F G;4G 8AFHE8 G;4G AB 74G4 <F ?BFG <9 4 546>HC .% G4>8FBI8E 49G8E 4 CE<@4EL .% 94<?F� ,;8A J8 78F6E<58 <A 78G4<? @4AL B9 G;8 CE46G<64? <FFH8F G;4G @HFG 58 477E8FF87 GB5H<?7 4 EB5HFG 6B@C?8G8 4A7 4HGB@4G87 FLFG8@� /8 4?FB78F6E<58 F8I8E4? 78F<:A 6;B<68F G;4G 4E<F8 9BE <@C?8@8AG<A:94H?GGB?8E4AG .%F 4A7 7<F6HFF G;8 GE478B�F <A G;8F8 6;B<68F�&8KG J8 :<I8 C8E9BE@4A68 E8FH?GF 9BE BHE <@C?8@8AG4G<BA9BE FB@8 58A6;@4E>F 4A7 FB@8 E84? 8AG8ECE<F8 4CC?<64G<BAF��<A4??L J8 78F6E<58 E8?4G87 JBE> 4A7 6BA6?H78�

2. BASIC FT DESIGN�<:HE8 � F;BJF G;8 54F<6 F8GHC B9 BHE FLFG8@ 9BE 94H?G

GB?8E4AG .%F� �BE 4 :<I8A .% 9BE J;<6; J8 78F<E8 GB CEBI<7894H?G GB?8E4A68 �G;8 -.'*�.4 .%� J8 EHA 4 �!(1- .% BA4 7<�8E8AG C;LF<64? F8EI8E G;4G <F >8CG <A FLA6 4A7 8K86HG8F<78AG<64??L GB G;8 CE<@4EL I<EGH4? @46;<A8 G;BH:; J<G; 4F@4?? G<@8 ?4:� /8 F4L G;4G G;8 GJB .%F 4E8 <A 2'.01�) ),!(�/0#-� ,;8 I<EGH4? 7<F>F 9BE G;8 .%F 4E8 BA F;4E87 FGBE4:8�FH6; 4F 4 �<5E8 �;4AA8? BE <+�+! 7<F> 4EE4L� 4A7 G;8E89BE8 4668FF<5?8 GB G;8 CE<@4EL 4A7 546>HC .% 9BE <ACHG 4A7BHGCHG� �/8 J<?? 7<F6HFF 4 78F<:A <A J;<6; G;8 CE<@4EL 4A7546>HC .% ;4I8 F8C4E4G8 ABAF;4E87 I<EGH4? 7<F>F <A +86G<BA ����� 'A?L G;8 CE<@4EL .% 47I8EG<F8F <GF CE8F8A68 BAG;8 A8GJBE> FB 4?? A8GJBE> <ACHGF 6B@8 GB G;8 CE<@4EL .%�+<@<?4E?L 4?? BG;8E <ACHGF �FH6; 4F >8L5B4E7 4A7 @BHF8� :BBA?L GB G;8 CE<@4EL .%��?? <ACHG G;4G G;8 CE<@4EL .% E868<I8F <F F8AG GB G;8

546>HC .% I<4 4 A8GJBE> 6BAA86G<BA >ABJA 4F G;8 ),%%'+%!&�++#)� �BE F8EI8E JBE>?B47F G;8 7B@<A4AG <ACHG GE4�6<F A8GJBE> 4A7 7<F>� �77<G<BA4? <A9BE@4G<BA 4F 7<F6HFF8758?BJ <A +86G<BA ��� <F GE4AF@<GG87 4F A868FF4EL GB 8AFHE8G;4G G;8 546>HC .% 8K86HG8F ABA78G8E@<A<FG<6 BC8E4G<BAF<A G;8 F4@8 J4L 4F G;8 CE<@4EL .%� ,;8 E8FH?G <F G;4G G;8546>HC .% 4?J4LF 8K86HG8F <78AG<64??L GB G;8 CE<@4EL .%� BJ8I8E G;8 BHGCHGF B9 G;8 546>HC .% 4E8 7EBCC87 5LG;8 ;LC8EI<FBE FB BA?L G;8 CE<@4EL CEB7H68F 46GH4? BHGCHGFG;4G 4E8 E8GHEA87 GB 6?<8AGF� �F 78F6E<587 <A +86G<BA ��� G;8CE<@4EL 4A7 546>HC .% 9B??BJ 4 FC86<�6 CEBGB6B? <A6?H7<A:8KC?<6<G 46>ABJ?87:@8AGF 5L G;8 546>HC .% <A BE78E GB8AFHE8 G;4G AB 74G4 <F ?BFG <9 G;8 CE<@4EL 94<?F�,B 78G86G <9 4 CE<@4EL BE 546>HC .% ;4F 94<?87 BHE FLF

G8@ HF8F 4 6B@5<A4G<BA B9 ;84EG584G<A: 58GJ88A G;8 E8?8I4AGF8EI8EF 4A7 @BA<GBE<A: B9 G;8 GE4�6 BA G;8 ?B::<A: 6;4AA8?�!A 477<G<BA J8 @HFG 8AFHE8 G;4G BA?L BA8 B9 G;8 CE<@4ELBE 546>HC .% G4>8F BI8E 8K86HG<BA 8I8A <9 G;8E8 <F 4 FC?<G5E4<A F<GH4G<BA J;8E8 G;8 CE<@4EL 4A7 546>HC F8EI8EF ;4I8?BFG 6B@@HA<64G<BA J<G; 846; BG;8E�!A G;8 9B??BJ<A: F86G<BAF J8 CEBI<78 @BE8 78G4<?F BA F8I

8E4? <@CBEG4AG 4E84F� !A +86G<BA ��� J8 :<I8 FB@8 78G4<?FBA G;8 78G8E@<A<FG<6 E8C?4L G86;AB?B:L G;4G 8AFHE8F G;4G CE<@4EL 4A7 546>HC .%F 4E8 >8CG <A FLA6 I<4 G;8 <A9BE@4G<BAF8AG BI8E G;8 ?B::<A: 6;4AA8?� !A +86G<BA ��� J8 78F6E<584 9HA74@8AG4? EH?8 B9 BHE �, CEBGB6B? G;4G 8AFHE8F G;4G AB74G4 <F ?BFG <9 G;8 CE<@4EL 94<?F� !A +86G<BA ��� J8 78F6E<58BHE @8G;B7F 9BE 78G86G<A: 4A7 E8FCBA7<A: GB 4 94<?HE8 <A 46BEE86G 94F;<BA�

2.1 Deterministic Replay Implementation�F J8 ;4I8 @8AG<BA87 E8C?<64G<A: F8EI8E �BE .%� 8K8

6HG<BA 64A 58 @B78?87 4F G;8 E8C?<64G<BA B9 4 78G8E@<A<FG<6 FG4G8 @46;<A8� !9 GJB 78G8E@<A<FG<6 FG4G8 @46;<A8F 4E8FG4EG87 <A G;8 F4@8 <A<G<4? FG4G8 4A7 CEBI<787 G;8 8K46G F4@8<ACHGF <A G;8 F4@8 BE78E G;8A G;8L J<?? :B G;EBH:; G;8 F4@8F8DH8A68F B9 FG4G8F 4A7 CEB7H68 G;8 F4@8 BHGCHGF� � I<EGH4? @46;<A8 ;4F 4 5EB47 F8G B9 <ACHGF <A6?H7<A: <A6B@<A:A8GJBE> C46>8GF 7<F> E847F 4A7 <ACHG 9EB@ G;8 >8L5B4E74A7 @BHF8� &BA78G8E@<A<FG<6 8I8AGF �FH6; 4F I<EGH4? <AG8EEHCGF� 4A7 ABA78G8E@<A<FG<6 BC8E4G<BAF �FH6; 4F E847<A:G;8 6?B6> 6L6?8 6BHAG8E B9 G;8 CEB68FFBE� 4?FB 4�86G G;8 .%�FFG4G8� ,;<F CE8F8AGF G;E88 6;4??8A:8F 9BE E8C?<64G<A: 8K86HG<BA B9 4AL .% EHAA<A: 4AL BC8E4G<A: FLFG8@ 4A7 JBE>?B47���� 6BEE86G?L 64CGHE<A: 4?? G;8 <ACHG 4A7 ABA78G8E@<A<F@A868FF4EL GB 8AFHE8 78G8E@<A<FG<6 8K86HG<BA B9 4 546>HC I<EGH4? @46;<A8 ��� 6BEE86G?L 4CC?L<A: G;8 <ACHGF 4A7 ABA78G8E@<A<F@ GB G;8 546>HC I<EGH4? @46;<A8 4A7 ��� 7B<A:FB <A 4 @4AA8E G;4G 7B8FA�G 78:E478 C8E9BE@4A68� !A 477<G<BA @4AL 6B@C?8K BC8E4G<BAF <A K�� @<6EBCEB68FFBEF ;4I8HA78�A87 ;8A68 ABA78G8E@<A<FG<6 F<78 8�86GF� �4CGHE<A:G;8F8 HA78�A87 F<78 8�86GF 4A7 E8C?4L<A: G;8@ GB CEB7H68G;8 F4@8 FG4G8 CE8F8AGF 4A 477<G<BA4? 6;4??8A:8�.%J4E8 78G8E@<A<FG<6 E8C?4L 1��3 CEBI<78F 8K46G?L G;<F

9HA6G<BA4?<GL 9BE K�� I<EGH4? @46;<A8F BA G;8 .%J4E8 I+C;8E8C?4G9BE@� �8G8E@<A<FG<6 E8C?4L E86BE7F G;8 <ACHGF B9 4 .%4A7 4?? CBFF<5?8 ABA78G8E@<A<F@ 4FFB6<4G87 J<G; G;8 .%8K86HG<BA <A 4 FGE84@ B9 ?B: 8AGE<8F JE<GG8A GB 4 ?B: �?8� ,;8.% 8K86HG<BA @4L 58 8K46G?L E8C?4L87 ?4G8E 5L E847<A: G;8?B: 8AGE<8F 9EB@ G;8 �?8� �BE ABA78G8E@<A<FG<6 BC8E4G<BAFFH�6<8AG <A9BE@4G<BA <F ?B::87 GB 4??BJ G;8 BC8E4G<BA GB 58E8CEB7H687 J<G; G;8 F4@8 FG4G8 6;4A:8 4A7 BHGCHG� �BEABA78G8E@<A<FG<6 8I8AGF FH6; 4F G<@8E BE !' 6B@C?8G<BA <A

31

VMM-based Primary Backup● Primary and backup execute on

two virtual machines

● Primary logs inputs and outputs● Backup applies inputs from log

● Primary-backup monitor each other◆ If primary fails, backup takes over

EECS 491 – Lecture 5 22September 17, 2019