boston university graduate school of arts and sciences

BOSTON UNIVERSITYGRADUATE SCHOOL OF ARTS AND SCIENCES

Dissertation

LOCAL ANONYMITY IN THE INTERNET

by

DAVID MICHAEL MARTIN JR.

B.S., Iowa State University, 1993

Submitted in partial fulfillment of therequirements for the degree of

Doctor of Philosophy1999

Committee Members

First Reader Steven Homer, Ph.D.

Professor of Computer Science

Boston University

Second Reader Sivaramakrishnan Rajagopalan, Ph.D.

Research Scientist

Bellcore

Third Reader Azer Bestavros, Ph.D.

Associate Professor of Computer Science

Boston University

c Copyright byDAVID MICHAEL MARTIN JR.1999

To private relationships

Acknowledgments

My advisor Steve Homer served as a reliable sounding board and guiding force in my studies at BostonUniversity. I am indebted to him for his unceasing support during my numerous transitions in Boston.

The seeds for this work were planted while I visited Bellcore’s Mathematics and Cryptology researchgroup in the summer of 1996 at Raj’s invitation. I thank him and Avi Rubin for working with me oninteresting problems.

I also thank Mason Katz at the University of Arizona, who was very helpful when I asked technical ques-tions about the x-kernel. Many others—in fact, everyone I approached—answered various little requests forclarification and updates very promptly. This is a fine reflection on the computer security community.

The Commonwealth research group got me thinking hard enough about systems issues that I couldreasonably contemplate this project, and it gave me as many opportunities for diversion as I wanted. I thankAzer Bestavros, Mark Crovella, and David Yates for organizing interesting projects, granting unlimitedaccess to the research group’s equipment, and financial support. Without those tangibles, the project wouldstill be an empty shell.

This work was partially supported by NSF research grants CCR-9706685 and CCR-9400229.On a more personal note, I am thankful for the friendships I developed with students at BU, especially

those I could almost claim to keep in touch with: Loredana Lo Conte, Laxmi Challa, Raj, Marcus Peinado,Bob Carter, Jun Liu, Paul Barford, and Drue Coles.

I continue to be grateful for the friendship and encouragement of Jim Lathrop, William Schmidt, andJack Lutz that began some years ago at Iowa State University.

I thank my parents Dave and Anne (I call them Mom and Dad) and my brother Hogan for continuing tocall me Davey even though I forbid others from doing so. I am sure everyone will agree that my relativesare better than theirs.

Finally, I thank Steve Godfrey for making me move before I was really ready. He knows that strongdeadlines motivate me well.

v

LOCAL ANONYMITY IN THE INTERNET

DAVID MICHAEL MARTIN JR.Boston University Graduate School of Arts and Sciences, 1999Major Professor: Steven Homer, Professor of Computer Science

Abstract

Packet-switched computer networks of all sizes are widely used for personal, professional, and governmen-tal communication. However, the speed, versatility, and largely unregulated nature of computer networkscoupled with the availability of affordable high-capacity storage makes it possible and even practical to rou-tinely maintain detailed logs of communication without indication to the communication partners. Althoughthe contents of messages can be protected by cryptography, the simple fact that the network carries an en-crypted message from one party to another is a piece of intelligence not as easily protected. For example,the message’s destination must be revealed at some level so the network can deliver it.

In this two-part dissertation we investigate techniques for protecting the identities of communicationpartners from worst-case adversaries such as the network infrastructure itself, including routers, switches,and hubs. In the first part, we survey and compare the nature of the untraceability offered by anonymity pro-tocols described in the literature. We then introduce local anonymity as a new model of network anonymitythat complements the protection afforded by existing network privacy schemes. In the second part, we de-scribe the experimental implementation of two protocols using local anonymity techniques and comparetheir performance. We find that while both protocols achieve satisfactory performance, superposed sending(also known as a DC network) offers better performance and security tradeoffs in most applications than thesimpler and more obvious alternative.

vii

Contents

Acknowledgments v

Abstract vii

List of Tables xv

List of Figures xvii

I The Structure of Anonymizing Networks 1

1 Introduction 31.1 Contributions of This Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

1.1.1 Main Result . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51.2 Thesis Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51.3 Thesis Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

2 Issues in Anonymizing Networks 72.1 Preliminary Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72.2 Protocols . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82.3 Definition of Anonymization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

2.3.1 The Meaning of “Endpoint” . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102.3.2 Deriving Anonymous Protocols . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

2.4 Adversaries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112.5 Endpoint-scrutinized Anonymity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122.6 Anonymized Protocols in a Network Stack . . . . . . . . . . . . . . . . . . . . . . . . . . . 122.7 Compatibility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142.8 Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152.9 Topology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

2.10 Administration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162.11 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

ix

3 Essential Techniques 193.1 The Ostrich Maneuver . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193.2 Randomized Routing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193.3 Crossing Administrative Domains . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203.4 Explicit and Implicit Addressing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213.5 Forwarding Through a Trusted Third Party . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

3.5.1 Responses in a Connection-Oriented Protocol . . . . . . . . . . . . . . . . . . . . . 213.6 Cryptography in Anonymizing Protocols . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

3.6.1 Sender-scrutinized Receiver Anonymity . . . . . . . . . . . . . . . . . . . . . . . . 223.6.2 Private Keys as Implicit Addresses . . . . . . . . . . . . . . . . . . . . . . . . . . . 223.6.3 Inheriting Low-level Anonymity . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233.6.4 Surface Characteristics Persist . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

3.7 Resisting the Forwarder Tracking Attack . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233.7.1 Redefining Protocol Semantics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233.7.2 Breaking the Timing Link . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243.7.3 Breaking the Size Link . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 253.7.4 Breaking the Existence Link . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

3.8 Cascades of Untrusted Forwarders . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263.8.1 Virtual Circuits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283.8.2 Layered Bidirectional Source Routing . . . . . . . . . . . . . . . . . . . . . . . . . 28

3.9 Resisting the Exclusion Attack . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 293.10 Mixes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 303.11 Superposed Sending . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 303.12 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

4 Prior Work 354.1 Anon.penet.fi . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 354.2 Cypherpunk Remailers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 354.3 The Mixmaster and Babel Remailers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 364.4 Nym.alias.net . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 394.5 The (Web) Anonymizer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 394.6 TAZ Servers and the Rewebber Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . 404.7 JANUS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 404.8 A Pedagogical Superposed Sending Network . . . . . . . . . . . . . . . . . . . . . . . . . 414.9 A Pedagogical Mix Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 424.10 Onion Routing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 424.11 Crowds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 434.12 The Non-Disclosure Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 454.13 The Lucent Personal Web Assistant . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 454.14 PipeNet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 474.15 Zero-Knowledge Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 474.16 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

x

5 Local Anonymity 515.1 Motivation and Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 515.2 Ramifications of Local Anonymity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52

5.2.1 Network Capacity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 535.2.2 Administration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 535.2.3 Inconspicuousness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 545.2.4 Responsibility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54

5.3 Applications of Local Anonymity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 565.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56

II Design and Implementation of the New Protocols 57

6 Introduction to the New Protocols 596.1 Protocol Basics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60

6.1.1 Outbound Packets: TO GW . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 606.1.2 Inbound Packets: TO LANON . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 606.1.3 Internal Authentication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 616.1.4 Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61

6.2 Shortcomings of the Protocols and Implementation . . . . . . . . . . . . . . . . . . . . . . 616.2.1 Inbound Packet Size Limitation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 636.2.2 Key Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 636.2.3 Membership Changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 636.2.4 Anonymized Network Layers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 636.2.5 Reliable Multicast Assumption . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 646.2.6 Multicast Limits Scalability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 646.2.7 Fault Tolerance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 656.2.8 Denial of Service Attacks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 656.2.9 Attacks Based on Other Characteristics . . . . . . . . . . . . . . . . . . . . . . . . 65

7 Experimental Platform Description 677.1 Computing and Networking Equipment . . . . . . . . . . . . . . . . . . . . . . . . . . . . 677.2 Operating System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 677.3 The x-kernel Networking Platform . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70

7.3.1 Threading in the x-kernel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 707.3.2 The Anonymizing IP Protocol Graph . . . . . . . . . . . . . . . . . . . . . . . . . 707.3.3 The Ethtap Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 707.3.4 Outbound Packet Processing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 737.3.5 Inbound Packet Processing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 737.3.6 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73

7.4 Experiment Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 747.4.1 The Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 757.4.2 Trace Gathering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 757.4.3 Post-processing and Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76

xi

8 Features Common to Both Protocols 798.1 Address Selection in External Packets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 798.2 Modifications to the Upper Layers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 808.3 Address Resolution Protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 808.4 Datagrams, Packets, Fragments, and Cells . . . . . . . . . . . . . . . . . . . . . . . . . . . 818.5 Gateway Contention . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 828.6 Types of Operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82

8.6.1 TRIGGER . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 828.6.2 POLL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 828.6.3 TO GW . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 838.6.4 TO LANON . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83

8.7 IPsec Authentication, Integrity, and Anti-Replay . . . . . . . . . . . . . . . . . . . . . . . . 848.8 Network Representation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84

9 The Trusted Gateway Protocol 859.1 Synopsis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 859.2 TO GW Operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85

9.2.1 TO GW BLATHER Operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 879.2.2 TO GW REPEAT Operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 879.2.3 Reacting to TO GW-type Operations . . . . . . . . . . . . . . . . . . . . . . . . . . 87

9.3 TRIGGER and POLL Operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 889.4 TO LANON Operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 889.5 Performance Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90

9.5.1 Scalability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 909.5.2 Contention Behavior . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 919.5.3 Indistinguishability Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . 939.5.4 Network Utilization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95

10 The Superposed Sending Protocol 9710.1 Synopsis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9710.2 Forwarding Topology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9710.3 Key-Sharing Graph . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9710.4 Key Material and Superposition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100

10.4.1 Arithmetic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10010.5 TO GW Operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100

10.5.1 Dummy Messages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10210.6 Collision Resolution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103

10.6.1 Strategy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10310.6.2 Leader Specification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10510.6.3 Internal Node Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10510.6.4 Collision Resolution Correctness . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109

10.7 TO LANON Operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11110.8 TRIGGER and POLL Operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11110.9 Performance Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112

xii

10.9.1 Scalability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11210.9.2 Contention Behavior . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11410.9.3 Indistinguishability Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11510.9.4 Response Time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11710.9.5 Network Utilization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12010.9.6 Key-Sharing Graph Complexity . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120

11 Conclusion 12311.1 Comparison of the Protocols . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12311.2 Future Directions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12411.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125

Bibliography 127

xiii

List of Tables

2.1 Properties characterizing an anonymized protocol . . . . . . . . . . . . . . . . . . . . . . . 17

3.1 Summary of Essential Techniques . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

4.1 Features of Related Projects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 484.2 Techniques Employed by Related Projects . . . . . . . . . . . . . . . . . . . . . . . . . . . 49

7.1 Computers Used in the Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69

10.1 One node’s perspective of the collision resolution process . . . . . . . . . . . . . . . . . . . 10710.2 Another node’s perspective . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10810.3 A longer collision resolution example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108

xv

List of Figures

1.1 The Secrecy Hierarchy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

2.1 An IEEE 802.3 (Ethernet) Frame . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82.2 Frame of Reference . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92.3 OSI and Internet Protocol Stacks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132.4 A slice of the network stack . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132.5 The network stack is reflected on the wire . . . . . . . . . . . . . . . . . . . . . . . . . . 142.6 Compatibility with Disclosed Protocols . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

3.1 Randomized Routing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203.2 The Forwarder Tracking Attack . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243.3 Eliminating Suspects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

4.1 Anon.penet.fi Operation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 364.2 Cypherpunk Remailer List . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 374.3 Cypherpunk and Mixmaster Remailer Operation . . . . . . . . . . . . . . . . . . . . . . . 384.4 An Encrypted Layered Path Encoded as a URL . . . . . . . . . . . . . . . . . . . . . . . 414.5 JANUS and Rewebber Operation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 424.6 Onion Routing Network Operation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 434.7 Crowds Network Operation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 444.8 Non-Disclosure Method Operation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

6.1 A Local Anonymizing Protocol Exchange . . . . . . . . . . . . . . . . . . . . . . . . . . 62

7.1 Physical Network Topology in the Experiments . . . . . . . . . . . . . . . . . . . . . . . 687.2 Anonymizing IP Protocol Graph, Top Half . . . . . . . . . . . . . . . . . . . . . . . . . . 717.3 Anonymizing IP Protocol Graph, Bottom Half . . . . . . . . . . . . . . . . . . . . . . . . 72

8.1 The Internet Protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81

9.1 A Trusted Gateway TO GW Operation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 869.2 The Trusted Gateway Header . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 869.3 The Trusted Gateway TRIGGER and POLL Operations . . . . . . . . . . . . . . . . . . . 899.4 The Trusted Gateway TO LANON Operation . . . . . . . . . . . . . . . . . . . . . . . . . 899.5 Trusted Gateway CPU Utilization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 909.6 Trusted Gateway Throughput . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91

xvii

9.7 Trusted Gateway Access Delay . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 929.8 Per-client Trusted Gateway Contention Throughput . . . . . . . . . . . . . . . . . . . . . 929.9 Aggregate Trusted Gateway Contention . . . . . . . . . . . . . . . . . . . . . . . . . . . 939.10 Trusted Gateway Contention Delays . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 949.11 Trusted Gateway Outbound Indistinguishability . . . . . . . . . . . . . . . . . . . . . . . 949.12 Trusted Gateway Inbound Indistinguishability . . . . . . . . . . . . . . . . . . . . . . . . 959.13 Trusted Gateway Network Utilization . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96

10.1 Superposed Sending Forwarding Topology . . . . . . . . . . . . . . . . . . . . . . . . . . 9810.2 Superposed Sending Key-Sharing Graph . . . . . . . . . . . . . . . . . . . . . . . . . . . 9910.3 Fragmentation Header . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10110.4 Collision Header . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10110.5 Superposed Sending TO GW Operation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10110.6 Superposed Sending Header . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10210.7 Leader Specification Pseudocode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10610.8 Internal Node Specification Pseudocode . . . . . . . . . . . . . . . . . . . . . . . . . . . 10910.9 Superposed Sending CPU Utilization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11210.10 Superposed Sending Throughput . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11310.11 Trusted Gateway Access Delay . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11510.12 Per-client Superposed Sending Contention Throughput . . . . . . . . . . . . . . . . . . . 11610.13 Aggregate Superposed Sending Contention . . . . . . . . . . . . . . . . . . . . . . . . . 11610.14 Superposed Sending Contention Delays . . . . . . . . . . . . . . . . . . . . . . . . . . . 11710.15 Superposed Sending Outbound Indistinguishability . . . . . . . . . . . . . . . . . . . . . 11810.16 Superposed Sending Inbound Indistinguishability . . . . . . . . . . . . . . . . . . . . . . 11810.17 Node 2’s Response Time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11910.18 Node 5’s Response Time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11910.19 Superposed Sending Network Utilization . . . . . . . . . . . . . . . . . . . . . . . . . . . 12010.20 Superposed Sending Key-Sharing Graph Complexity . . . . . . . . . . . . . . . . . . . . 121

xviii

Part I

The Structure of Anonymizing Networks

Chapter 1

Introduction

Anonymity is not the same as confidentiality. Existing systems such as IPsec [5] can ably protect the contentsof IP-based communication, making it impractical for network eavesdroppers to deduce the purpose ofobserved traffic. However, the fact of the traffic itself is not hidden; the sender and receiver of each packet isclearly visible in the packet headers even when the packet is end-to-end encrypted. Such information shouldsometimes be protected as well.

Example 1. Alice, the CEO of Apricot, decides to buy out the publicly-traded firm Banananon. Whenother Apricot officers learn of the plans, the subsequent flurry of connections from the executive suite tothe Banananon’s web site could betray Alice’s intent. If Alice’s network administrator Eve reacts to thenetwork connection logs by purchasing Banananon stock, then Alice and others could end up in serioustrouble. Anonymity schemes can prevent Eve from identifying the executive suite as the source of thetraffic, even though the traffic flows through Eve’s equipment.

Example 2. Wartime commander Adam is issuing orders for troop movements by satellite. Adam is an ex-tremely valuable tactician, and his superiors fear that his enemies plan an assassination. They therefore seekan anonymity scheme that keeps his current location secret while allowing him to continue communicating.(This example is due to [32].)

Example 3. Adrienne works at a U.S. nuclear plant and suspects irregularities in her administration’swaste disposal procedures. She decides to contact the Nuclear Regulatory Commission to investigate herreporting options. But she stops while typing in the secure URL https://www.nrc.gov, realizing thather boss can still see “www.nrc.gov” just by watching her network connection.

Example 4. Andrew is an outspoken moral crusader about to undergo confirmation for his appointmentto federal court. One afternoon he is surfing the web for information about fraud, “hustling”, and other congames. Suddenly, a misleading link sends him deep into the hierarchy of http://www.hustler.com.The next day, Larry Flynt [48] calls a press conference. Could this have been prevented?

Example 5. The Civil Liberties Committee of the European Parliament released a report in Decemberof 1997 describing Echelon, an alleged National Security Agency (NSA) project that “routinely and indis-

4 Chapter 1. Introduction

? ?

Secrecy of Secrecy

Secrecy ofConnection

Secrecy ofContent

Content

Steganography,Inconspicuousness

Anonymity

Encryption

Cleartext

Incr

easi

ng S

ecre

cy

Incr

easi

ng E

xpen

se

Problem Solution

Figure 1.1: The Secrecy Hierarchy. Anonymity can be thought of as a higher-ordersecrecy problem.

criminately” monitors phone, fax, and email messages from spy stations distributed throughout the world[27, 28]. Echelon is said to monitor mostly non-military targets, including governments and businessesaround the world. When the adversary is everywhere, does privacy stand a chance?

This thesis examines techniques for achieving anonymity in the Internet. The problem is challengingbecause the quantities used to identify communication partners—the IP addresses—are also used to routemessages. Anonymity is essentially the separation of the identification function from the routing function.

A number of Internet anonymizing schemes have been proposed or implemented in the past severalyears, differing widely in application (see Chapter 4). Some anonymize email, some anonymize web traffic,and some claim general applicability. Particularly interesting is the fact that none of the built anonymizersrely on any other anonymizing technology: they are all “stand-alone,” typically customized to work onspecific application protocols that are constructed on top of TCP.

This work is distinguished by its concentration on anonymity as a primitive. We have no single ap-plication in mind. Instead, we develop protocols for IP whose anonymizing properties are at least partiallyinheritable by higher level protocols. In other words, our protocols are intended to be used to construct moresophisticated anonymizing schemes. As a result, our protocols may be composed with application-orientedanonymizers without undue effort.

We treat anonymity from both security and networking perspectives. Within security, anonymity canbe viewed as meta-secrecy, that is, secrecy of the observable part of a traditionally secured communicationstream (see Figure 1.1). We develop protocols that attempt to minimize the ability of a passive or activeadversary to penetrate this meta-secrecy.

While one of the protocols we develop depends on a technique that has been proven secure underinformation-theoretic assumptions, we do not prove or claim reductions to our computational environment.The anonymity properties of our protocols—i.e., the security “results”—are strong in only an intellectuallyappealing sense. However, our protocols are designed to be among the least vulnerable to attack of allpublished to date.

Chapter 1. Introduction 5

Viewing anonymity as a networking topic, we consider issues such as throughput, delay, scalability,reliability, application interface, and routing. Of these, we concentrate on performance and scalability in theprotocols we develop.

1.1 Contributions of This Work

We claim four main contributions of this work:

� We survey anonymizing techniques and classify existing anonymizing technologies with one voice.

� We articulate a combination of untraceability goals, realistic computing infrastructures, and socialenvironments that we believe are pervasive enough to warrant a name: “local anonymity.”

� We describe the design and implementation of two IP anonymizing protocols that are consistent withthe local anonymity model and give detailed performance results in a realistic network environment.Our experiments demonstrate that IP flows can be anonymized in our model at a tolerable cost withreasonable performance using off-the-shelf computing and networking hardware.

� One of our protocols contains the first application of the Superposed Sending technique (also andoriginally known as the Dining Cryptographer technique) to real network protocols. We describeour optimizations and implementation choices, and show why this technique is most appropriate forheterogeneous network and computing environments.

1.1.1 Main Result

Of these contributions, the main result of this work is proof by exhibition that high-quality anonymity canbe achieved within a local area network using current (ca. 1998) commodity technology. We describe indetail a 100 Mbit/sec Ethernet LAN of 12 PC nodes whose communication endpoints are less traceablethan any other published protocol implementation, yet the LAN can still sustain outbound throughput of 2Mbit/sec and inbound throughput of 18 Mbit/sec. We also present results that address outbound scalabilityfor networks of up to 16 nodes.

1.2 Thesis Scope

Notwithstanding our survey of issues, essential techniques, and related work in Chapters 2 through 4, thisthesis focuses on the problem of local (LAN) anonymity. We argue first that local anonymity is useful, andthen that it is achievable. We do not project our results onto larger structures such as anonymous networks ofanonymous LANs. A. Pfitzmann’s dissertation [67] presents hierarchical designs for large scale anonymousnetworks built from protocols such as those developed here.

1.3 Thesis Outline

This thesis is presented in two parts. In the first part, we investigate the structure of anonymizing networksin general terms. We begin in Chapter 2 by defining anonymity and listing the properties that are relevantwhen evaluating an anonymizing scheme.

6 Chapter 1. Introduction

In Chapter 3, we describe the essential techniques available to a designer of anonymous protocols inorder of increasing complexity. Meanwhile, we explain attack scenarios that motivate the increasinglycomplex and expensive techniques.

We proceed in chapter 4 by investigating other network anonymity research projects in terms of therelevant issues and essential techniques developed in chapters 2 and 3. We concentrate on projects that havereached at least the prototype stage. However, some undeployed projects describe techniques that we findsufficiently promising to warrant attention as well.

In Chapter 5 we conclude Part I with a description of the “local anonymity” approach. Local anonym-ity is not a new technical means of achieving anonymity but rather the solution to a scenario combiningparticular anonymity wishes, available computing resources, and social structures that we find unaddressedby existing anonymizing technologies. Local anonymity can be seen simply as a combination of essentialtechniques that make sense in such situations.

In Part II of this thesis, we take the local anonymity concept to fruition by describing a design andimplementation of the concept using two competing protocols. We introduce the implementations and con-vey the protocols’ anonymity properties in general terms in Chapter 6. In addition, we itemize a numberof shortcomings of the protocols and our implementation in Chapter 6; our implementation work has onlyreached the prototype stage.

Chapter 7 contains a description of the hardware and software platform we used for our implementa-tion. The type of computing and networking equipment we used is really only relevant to put the perfor-mance results of later chapters in perspective. But the prototype software architecture—particularly the OSinterface—limits our ability to extend anonymity to higher-level protocols such as TCP. Also in Chapter 7,we describe the basic set of experiments that we ran on each protocol to produce performance results.

In Chapter 8, we cover issues that are treated similarly in the two protocols. Since they are both instancesof local anonymity, their principles of operation are similar. And since both protocols are realized in a singleimplementation, much of the code was reused.

We describe the Trusted Gateway protocol in Chapter 9: its specification and behavior, packet contents,and performance in our test environment.

Chapter 10 follows the same pattern for the Superposed Sending protocol. Since this protocol is not astransparent as the Trusted Gateway, we explain its behavior in more detail, providing example interpretationsand pseudocode specifications.

We conclude this thesis with a brief interpretation of the comparative performance results and a list ofsuggestions for future work.

Chapter 2

Issues in Anonymizing Networks

In this chapter we define what is meant by anonymizing networks and construct a taxonomy of their charac-teristics.

2.1 Preliminary Definitions

First, we use bold face when introducing a term. Definitions that are uncommon or particularly importantare broken out of the text for visibility.

We use the generic terms computer, workstation, and (network) node interchangeably to denote anycomputer, not necessarily one specifically deployed as a networking infrastructure component. A gatewayis a node that translates data between multiple network protocols.

Throughout this text we concentrate on the Internet Protocol version 4 (IP) [73] and the related ICMP[72] (Internet Control Message Protocol), TCP [74] (Transmission Control Protocol), and UDP [71] (UserDatagram Protocol), although our results may apply to other stateless and unreliable network protocols withcomparable addressing characteristics. See [90] or [19] for a comprehensive treatment of the IP protocolsuite.

A network message is any individual communication carried on a computer network. A cell is a fixed-length sequence of (usually encrypted) bytes. A frame is a variable-length sequence of bytes formatted fortransmission on a physical network such as Ethernet [44]. See figure 2.1. A packet is a variable-lengthsequence of bytes formatted for encapsulation within a frame. Since our research concentrates on the IPlayer, we devote most of our attention to IP packets. The IP packet format is shown in Figure 8.1 onpage 81.

A hub is a device that logically ORs multiple subnets of a broadcast network so that each attachedsubnet experiences the aggregate network traffic. A (network) switch is a multiport device that routesframes independently from one of its ports to another according to their lowest-level addressing, therebyshielding other attached nodes from irrelevant traffic. Switches generally support broadcast and multicastaddressing modes as well. We consider switches, hubs, wires, and “off-the-shelf” higher level devicesincluding bridges and routers components of the networking infrastructure.

Suppose that node A sends a packet to node B. Fixing the frame of reference at node A, we say thatthe packet is outbound, and any reply from B is inbound. Other nodes that are near A in a networking oradministrative sense are considered interior nodes, while others are exterior. (The meaning of “near” will

8 Chapter 2. Issues in Anonymizing Networks

Preamble

payload + pad38-1500 bytes

start Dest

AddrSrcAddr

length

Checksum

0 8 16 21

0 8 16 21

Figure 2.1: An IEEE 802.3 (Ethernet) frame. The scale is in bytes. The minimumsize of a frame is 64 bytes.

be made clear when such terms are used.) See Figure 2.2.A bidirectional communication channel such as TCP is called a connection. A connection involving

node A is outbound if A is the connection initiator, otherwise it is inbound and A is the connectionresponder. In the language of TCP, the initiating party is said to perform an active open and the respondingparty performs a passive open. Note that every nontrivial connection carries both inbound and outboundpackets.

2.2 Protocols

Most of this text concerns the behavior of protocols. Informally, a network protocol is a specificationof a distributed algorithm used to carry information between nodes on a local area or wide area network.Real-world examples range from the low-level MAC protocol in the IEEE 802.3 standard [44] (we refer tothis as the Ethernet protocol) to the high-level Hypertext Transfer Protocol (HTTP) [36]. The algorithmspecified by a network protocol consists of two alternating types of operations: (1) purely local computationthat proceeds independent of attached network activity, and (2) network operations that provide access tothe attached networks.

The unadorned term “protocol” sometimes refers to a program that effects communication in a mannerconsistent with a (protocol) specification. This is more properly called an implementation or realizationof a protocol.

2.3 Definition of Anonymization

In conventional network cryptography, one party A decides to send a message M to another party B. Toproceed, A protects M with a secret key k, producing a ciphertext k(M). The names A and B and thedirection of the communication are generally explicit, so the message would appear on the wire as A! B :

k(M). Note that the message header A! B is unencrypted, so the communication endpoints are visible toall observers. (The use of authentication within M to resist active attacks against the message header suchas address spoofing is well understood [82, 57, 46].) To be concerned with network anonymity is to keepsuch message headers secret.

Chapter 2. Issues in Anonymizing Networks 9

`

BA

outbound message

internal nodes

external nodes

inbound message

Figure 2.2: Frame of reference about node A.

Definition. An anonymizing protocol is a network protocol that meaningfully restricts an adversary’s abil-ity to infer the true endpoints of a network communication. An anonymizing network is a network of nodesthat communicate using anonymizing protocols.

Suppose that A decides to send an anonymized message to B.

� If the adversary is unable1 to conclude that the message source is A, then the anonymizing protocol isproviding origin or sender anonymity.

� If the adversary is unable to conclude that the message destination isB, then the anonymizing protocolis providing destination or receiver anonymity.

� Given multiple anonymized messages from A toB, if the adversary is unable to determine whether themessages refer to the same same source and destination, then the anonymizing protocol is providingsender or receiver unlinkability. Unlinkability is stronger than sender or receiver anonymity: notonly is the adversary unable to correctly identify any single message’s endpoint, but the adversary isalso unable to determine a renaming of endpoints consistent with a sequence of observed messages.

The following terms are not in standard use, but we are unaware of any standard equivalents.

Definition. A network protocol that is not anonymizing is called a disclosing protocol.

Definition. A network or node that is the object of an adversarial attempt to determine communicationendpoints is a scrutinized network or node.

1Indefinite terms such as “unable to conclude” and “identify” depend on the particulars of an anonymizing protocol. In generalit is safe to think of the restrictions as increasing the adversary’s probability of error.


2.3.1 The Meaning of “Endpoint”

In this work, “knowing the true communication endpoint” means knowing a valid network address thatuniquely identifies a node. It makes sense to consider the confidence of such knowledge under this def-inition, but we do not stray from the fundamental association between “identity” and “network address”.Epistemological issues are left for those who must subsequently interpret identity assertions emerging fromanonymizing or identifying technologies.

For example, an identifying technology might produce evidence that node A sent message M . Althoughthe owner of A might go on to assert that it simply relayed M from elsewhere and did not invent it, thetechnologies we investigate are not able to support or refute such claims.

As another example, a technology for receiver anonymity might cause a message M to be sent to nodes1 : : : 10 while only nodes 3 : : : 7 actually process M and the rest simply drop it. One might want to think ofthe collection 3 : : : 7 as the protected identity, like the membership of a secret committee [9]. But this is akind of meta-address, not a single node’s network address, and so we do not address it in this work.

2.3.2 Deriving Anonymous Protocols

Anonymizing protocols are really only interesting when they do something besides anonymize; otherwise,no useful information is exchanged and there is no need for a protocol at all. For this reason we concentrateon anonymized versions of useful disclosing network protocols. That is, anonymizing protocols are derivedfrom disclosing protocols in this text.

For instance, an anonymizing protocol may be derived by encrypting and encapsulating a disclosedprotocol (such as NFS) inside headers that somehow keep identities secret. This approach preserves the se-mantics of the disclosed protocol without requiring knowledge of them, but it also largely preserves surfacecharacteristics of the disclosed protocol such as message length and message frequency. These character-istics may be useful to an attacker and are discussed in x3.6.

Most disclosed protocols provide for two-way communication between peers; one party initiates a com-munication, and the other responds. An anonymous version of such a protocol will incorporate techniquesfor both sender and receiver anonymity from different perspectives. We call it sender anonymity when Aliceanonymously sends to Bob, but when Bob replies to Alice, the protocol must provide receiver anonymity toAlice—otherwise, her identity is trivially betrayed.

To clarify this situation we define two more terms.

Definition. An anonymous client protocol is one that protects the identity of a client by sender anonymitywhen the client transmits to its peer and receiver anonymity when the client receives from its peer. Forexample, an anonymous client protocol would keep the identity of a web browser node secret as it accessedweb sites, but the identity of the web sites the client accessed would not necessarily be protected.

Definition. An anonymous server protocol is one that protects the identity of a server through applicationof receiver anonymity for messages directed to the server and sender anonymity for the server’s messagesto the client. An anonymous server protocol could be a site providing web service without betraying its ownidentity.

Similarly, one can define an anonymous rendezvous protocol as one that allows for communicationbetween a client and a server without betraying the identity of either endpoint. An anonymous rendezvous


protocol could be used in an auction where buyers and sellers want to remain anonymous in the negotiatingstages and reveal their identity to each other after a deal has been struck. We do not pursue this notionfurther and instead concentrate on anonymous sender, client, receiver, and server protocols.

2.4 Adversaries

To understand the protection offered by an anonymizing protocol, we turn our attention to the abilities oftwo types of adversaries also known as attackers. Passive adversaries are able to examine network trafficonly, while active adversaries are able to modify network traffic as well. However, we assume that no activeor passive adversary is able to penetrate a scrutinized node beyond the external side of its network interface.In other words, these adversaries may only intercept or perturb type (2) protocol operations defined in x2.2.An active attacker may be able to rewrite every bit of data that a scrutinized node reads from or writes to thenetwork, but it may not directly access the node’s state.

We occasionally refer to node infiltration by an adversary. In this case the adversary has completelytaken over the node and is able to read and write the node’s memory in addition to all of the abilities of anactive attacker at that network location. Resisting an adversary present in infiltrated nodes is more difficultthan resisting active attacks because the infiltrated nodes may have access to cryptographic key materialand other secrets not directly available to weaker attackers. Nonetheless, we will investigate anonymizingtechniques that resist infiltration of certain nodes as well.

While the adversary always aims to determine a network address, the the scrutinized party’s degree ofanonymity can vary. The scrutinized party’s degree of anonymity varies inversely with the confidence in theadversary’s identity conclusion. For instance, the adversary may be seeking simple statistical evidence ofa transgression, which if found might lead to a bawling out of the staff without singling out any employee;or, the employer might use the evidence as motivation to install video surveillance equipment. On the otherhand, the adversary may be looking for one incontrovertibly correct bit of information to use in a court case.This degree of anonymity notion is due to Reiter and Rubin [80].

We usually refer to an adversary in the singular even though the adversary may actually consist of aset of cooperating nodes. In this case we more accurately call it a multi-node or distributed adversary.While an anonymizing protocol can avoid many single-node adversaries just by forwarding through anaccomplice (see x3.5 and x3.6), remaining anonymous against a coordinated and distributed adversary canbe a formidable challenge (x3.7).

Note that protocols can anonymize only at a level commensurate with the adversary’s initial searchspace. For example, if a network protocol allocates address space for only two source addresses (admit-tedly a rather lacking design), then it must be assumed that the adversary’s probability of message sourceidentification is at least 1/2. The same probability holds even if the protocol’s address space is large but theadversary is assumed to know that only two nodes in the world use the protocol. Therefore cognizance ofthe adversary’s initial search space is an important parameter when evaluating an anonymizing protocol.

These considerations also allow us to conclude that a meaningfully anonymizing protocol always routesanonymized messages through additional anonymizing nodes on their way to their peers. Otherwise theadversary will not care or even notice that the single node—the obvious communication endpoint—is at-tempting to anonymize anything.


2.5 Endpoint-scrutinized Anonymity

An anonymous protocol offers protection in at least one of the three basic categories defined above (senderanonymity, receiver anonymity, unlinkability). However, this designation may not adequately describe thethreat countered by the protocol. We may need to specify the scrutinized party being protected by applica-tions of the basic properties as well.

For example, suppose Alice is interested in hiding the fact of her communication with Bob. This couldmean at least two things: first, she might want an anonymizing protocol to provide sender anonymity sothat an external adversary could not conclude that she was communicating (with anyone). Second, shemight want an anonymizing protocol to provide receiver anonymity to Bob so that an internal adversarycould not conclude that she was communicating with Bob, even if the adversary could see some kind ofcommunication. In the latter case, Bob’s address is hidden but Alice benefits; we call it sender-scrutinizedreceiver anonymity. We define the converse property receiver-scrutinized sender anonymity similarly.

2.6 Anonymized Protocols in a Network Stack

The multiple protocols used to communicate a message between nodes are usually arranged in a stack calleda protocol stack or network stack [94]. The lowest levels of the stack are used to communicate the mostphysically-oriented information (such as electrical characteristics and bit framing), while the highest levelsof the stack are used to carry the most application-specific information.

For example, consider the ubiquitous Sockets/TCP/IP/Ethernet stack. At the bottom, Ethernet is a low-level, unreliable, connectionless protocol. However, through the maintenance of state at the IP, TCP, andSockets layers, Ethernet can ultimately carry the high-level, reliable, connection-oriented protocol HTTPbetween a web client and server. This arrangement is reflected both by the order that the protocol-specificdata appears on the physical network and by the architecture of the networking implementation itself. SeeFigures 2.3, 2.4 and 2.5.

Since internetwork protocols are usually constructed as part of a protocol stack, the question of whichprotocols to anonymize becomes particularly important. As a general rule, low-level protocols in a networkstack are more widely useful when anonymized than high-level protocols, because high-level protocols areencapsulated in low-level ones and so they automatically take on a measure of the low-level protocol’sanonymity protection. However, different layers of the stack use different addressing conventions. Underthe assumption that the high-level address is the object of attack, simply encapsulating it in a protocol thatprotects only low-level addresses does not solve the problem. In other words, it does little good to anonymizeEthernet headers alone if the goal is build an anonymous TCP/IP connection, since an adversary could learnthe endpoint IP addresses by ignoring the Ethernet headers and examining the IP headers directly. We revisitthis issue in x3.6.3.

On the other hand, anonymizing high-level protocols in isolation can be problematic as well. A resource-ful adversary may be able to observe a series of low-level point-to-point messages and infer a high-levelprotocol connection as a result. This does not render the anonymizing high-level protocol useless, but itdoes affect the type of adversaries that the anonymizing protocol can withstand.

In addition, low-level network protocols are often implemented in highly optimized hardware. Equip-ment costs alone may make it prohibitively expensive to anonymize the lowest level protocols. At the pointwhere an adversary relies on electrical signal analysis to locate transmitters on a single network, the question


Application

Presentation

Session

Transport

Network

Data Link

Physical

Application(HTTP, telnet, ...)

Transport(TCP, UDP...)

Internet(IP)

Physical(Ethernet, FDDI, ...)

Figure 2.3: OSI (left) and Internet (right) protocol stacks. This reckoning is due toTanenbaum [94], who also provides a historical perspective on the twomodels.

Protocol Name Protocol Header Implemented inHTTP GET / HTTP/1.0 : : : ApplicationTCP port 0B21 ! 0050 OS

IP82.FD.C0.E2 !

C7.B5.AC.F7OS

Ethernet00:10:4B:65:9A:AF !

00:00:0C:1A:36:EFNetwork interface

Figure 2.4: The high-level protocol HTTP is carried by lower-level protocols. Ac-cordingly, high-level application code initiates the network request anddeposits the HTTP headers onto an outbound message. The lowest-level networking code adds the lowest-level headers right before trans-mitting the message.


0000 0C1A 36EF 0010 4B65 9AAF 0800 4500 .....6...Ke....E.022A 36A9 4000 4006 4998 82FD C0E2 C7B5 ..*6.@[email protected] 0B21 0050 C2D8 A9F6 61F5 5AF0 5018 ....!.P....a.Z.P.7D78 D0F7 0000 4745 5420 2F20 4854 5450 .gx....GET / HTTP2F31 2E30 0D0A 436F 6E6E 6563 7469 6F6E P/1.0..Connection3A20 4B65 6570 2D41 6C69 7665 0D0A 5573 n: Keep-Alive..Us...

Figure 2.5: The network stack is reflected on the wire. Each of the relevant protocolheaders from Figure 2.4 appears here (in bold font) as it was capturedfrom the network, with the lowest-level protocol appearing first.

of anonymity has left the realm of protocols and has entered electrical engineering.Therefore, we build anonymity on top of standard link-level protocols (Ethernet, FDDI, SONET, etc.) in

this work. This implies that the link-level addresses are not the protected addresses, since they are explicitlyrevealed on the wire. A node that wants to know who transmitted a frame on the LAN can simply consult itsnetwork interface to find out. But the network interface does not know the whole route that the frame tookin order to get there. The anonymity of higher-level protocol addresses will come from this uncertainty.

2.7 Compatibility

While we would like to provide a measure of anonymity without significantly restricting the space of reach-able nodes, we consider the prospect of a large-scale anonymizing renovation of the Internet unlikely. Thefact that IPv4-based protocols are routinely used in spite of their lack of security mechanisms (authentica-tion, encryption, anti-spoofing) is ample evidence that even major security improvements such as IPSEC arenot yet considered important enough to significantly affect the installed base and market. Therefore in ourfocus on anonymizing versions of Internet protocols there is an implicit assumption of compatibility withthe existing (disclosing) protocols.

In general the party that wishes to remain anonymous will be the direct user of the anonymizing protocoland the peer will run the disclosing version. This puts the anonymity burden on the party that benefits andrequires nothing additional of the peer. For example, a scrutinized sender would begin a transaction usingthe anonymizing protocol, but the receiver would use the disclosing protocol. Let peer refer to a sender, areceiver, a client, or a server. An anonymizing protocol is said to be peer-compatible if the anonymizingprotocol can communicate directly with peer’s (unmodified) disclosing protocol. Thus client anonymity isusually server compatible while receiver anonymity is usually sender compatible.

This arrangement implies the existence of a gateway between the disclosing and anonymizing version ofthe protocol. Protocol gatewaying is a function that is not necessarily fixed in a spatial or temporal sense. Anetwork may have many gateways in many locations over time. Figure 2.6 shows a fixed gateway that servesa small anonymizing network. The part of the network between the anonymous gateway and the disclosingpeer runs the disclosing protocol and may be vulnerable to both passive and active attacks. Consequently,the placement of the gateway directly affects the adversary’s initial search space: the space can be no largerthan the network hidden “behind” the anonymous gateway, i.e., the network connected to the gateway thatoffers resistance to the adversary.

Many of the techniques that we discuss in this text are easy to envision in a purely anonymous environ-


disclosingprotocol

anonymizingprotocol

gateway

Internet

Figure 2.6: Compatibility with disclosed protocols. The anonymizing networkwithin the cloud runs the anonymizing protocol, but its fixed gatewayconverts between the anonymizing protocol and the disclosed protocolfor communication with nonmember nodes.

ment, that is, one with no compatibility support and therefore no gateway to a disclosed protocol. Such anenvironment allows us to consider the possibility of anonymous rendezvous protocols that offer resistance toa distributed adversary located anywhere along the route connecting the endpoints. However, the possibilityof deadlock and starvation in this scenario is particularly vexing, because deadlock prevention and recogni-tion techniques require the reliable identification of resources. Although the identifiers can be arbitrary aslong as they are consistent, their exposure as a deadlock countermeasure could be a great aid to an adversaryattempting to learn the relationships between scrutinized nodes.

2.8 Performance

The performance of network protocols is primarily characterized by three quantities: the throughput theyprovide for useful communication, the bandwidth they consume from the underlying network, and thedelay or latency they impose on useful communication. Network utilization is a derived metric defined asthe ratio of bandwidth consumption to bandwidth available.

It is extremely difficult to precisely describe the performance of a protocol, since this amounts to min-imizing the performance parameter among all possible implementations. However, the performance of aparticular protocol implementation is easily sampled, and if the implementation is reasonably thought to begood, the sample can be taken as an approximation of the protocol’s performance. In order to understandthe performance of a network protocol implementation, we also examine standard computing metrics suchas CPU utilization and task delay.

An anonymized network protocol will have different performance than the disclosing version of thesame protocol. If the gap is significant across a performance parameter, then this may change the natureand usefulness of the protocol. For instance, one simple way to anonymize HTTP on a multiuser systemwith many web browsers is for the OS to intercept all outbound HTTP connection requests and send themall out en masse on the hour (see x3.7.2). That would make it difficult for an adversary to correlate anindividual outbound connection with a particular web browser user. But this would have a drastic effect on


the behavior of the WWW as seen by the web browser users. HTTP works at a much smaller granularitythan web pages; each page usually requires several HTTP transactions. So visiting a single page under thisscheme could keep a user waiting for hours. At the very least, if a disclosing protocol is interactive, then ananonymizing version of it should be interactive as well. This rule of thumb influences the design of most ofthe anonymizers described in Chapter 4.

In chapters 9 and 10, we analyze the performance of our implementation of two new protocols. Asparameters of the experiments (such as network size and offered contention) are varied, the relative perfor-mance results confirm our expectations. But the absolute performance results are also interesting, as they gostraight to the usability of the protocols.

2.9 Topology

Since the message headers protected by an anonymizing protocol are generally used for message routing, theintroduction of a anonymizing protocol may presuppose a certain type of network topology. For example,receiver anonymity is not difficult to achieve under the assumption that the intended receiver is reachableonly by satellite broadcast. Thus topology constraints can be considered another restriction on the applica-bility of an anonymized protocol. They can also be thought of as performance requirements of the networklayer below the anonymizing layer, since performance is a primary consequence of a network topology.

2.10 Administration

There may be administrative overhead involved in deploying an anonymized protocol. Typical issues in-clude address assignment, cryptographic key exchange and certification, resource allocation, fault reportingand recovery (troubleshooting), and performance tracking. These areas are likely to be affected by theanonymization of a protocol, since this necessarily obscures the identity of participants.

2.11 Summary

In this chapter we defined terms relevant to anonymized networks and itemized facets of their design thatallow us to differentiate between anonymizing protocols. Figure 2.1 summarizes these properties.


PROPERTY EXAMPLES

TYPE OF ANONYMITYSENDER, RECEIVER, UNLINKABILITY

CLIENT, SERVER, RENDEZVOUS

SCRUTINIZED PARTY SENDER, RECEIVER

COMPATIBILITYSENDER, RECEIVER

CLIENT, SERVER

ADVERSARIES RESISTEDACTIVE, PASSIVE

INTERNAL, EXTERNALINITIAL SEARCH SPACE

LOCATION IN STACK LOW LEVEL: CLOSE TO HARDWAREHIGH LEVEL: CLOSE TO APPLICATION

PERFORMANCE IMPACTUSEFUL THROUGHPUT

CONSUMED BANDWIDTHLATENCY

TOPOLOGY CONSTRAINTS BROADCAST, STAR, SWITCH

ADMINISTRATIONCONFIGURATION, REPORTING,

TROUBLESHOOTING

Table 2.1: Properties characterizing an anonymized protocol

Chapter 3

Essential Techniques

As noted previously, any useful anonymizing protocol assumes the existence of multiple anonymizing nodes.However, there is considerable flexibility in deciding what to do with these nodes. In this chapter we surveyand describe properties of known anonymizing techniques, roughly in order of increasing complexity. Atthe same time, we introduce known avenues of attack.

All of the techniques in this chapter were either described or implied by existing work and are morefully explored elsewhere; our purpose is only to classify them in broad strokes so that the techniques andproperties of local anonymity presented in Chapter 5 and Part II can be put in context. We therefore presentthe techniques below in general terms and avoid concentrating on specific applications. The single mostuseful exposition of anonymity and reference on the techniques is certainly A. Pfitzmann’s dissertation [67];unfortunately, there is currently no English translation available.

3.1 The Ostrich Maneuver

Most network protocols do not provide for anonymity. In the absence of an adversary this is adequate, sincethere is no one there to exploit the situation. Unfortunately we usually do not know when an adversary ispresent.

The simplest attack on a disclosing protocol is a network sniffer located somewhere along a message’sroute. The adversary need only read the intercepted messages to learn the communication endpoints. Totrace a single node’s communication, the adversary should be placed as close to that node as possible.Similarly, a network’s internet communication can be intercepted by placing the adversary at the LAN-WAN border.

3.2 Randomized Routing

Randomized routing is a very general technique that can be used to minimize exposure to adversaries fixed innetwork space, as depicted in Figure 3.1. We assume that the scrutinized party Alice wants to communicatewith one of multiple designated anonymizing nodes but Alice does not know where the adversary is presentin the network. (If Alice knew where a specific attacker Eve was present, then Alice could attempt to simplyroute around Eve.) By choosing between anonymizing nodes randomly, the immobile adversary only comesin contact with a fraction of the offered traffic.

20 Chapter 3. Essential Techniques

1

2 3

7

4 5

6

Adversary

Figure 3.1: Illustration of randomized routing. The adversary is fixed in networkspace with direct access to all communications involving nodes 3 and7. By choosing forwarding paths randomly, the other nodes have agood chance of avoiding the adversary altogether.

3.3 Crossing Administrative Domains

An administrative domain1 is a contiguous network under the control of a single authority. For example,network administrators usually directly manage only within their domain and make phone calls or other out-of-band contact with other authorities as their responsibilities require. The authority in question need not bea single person; it could be a committee or other distributed entity. In practical terms one would considertwo authorities to be in separate administrative domains if both would plausibly deny the possibility that theother could successfully demand their complicity in eavesdropping or other invasive pursuits.

It is tempting to consider the adversary to be spatially restricted to the partition defined by the adminis-trative boundaries. It is certainly harder for an adversary to be distributed across administrative borders thanto be distributed within a single administrative domain. Therefore, we can expect that anonymous trafficthat crosses administrative domains exposes less of the protocol to an adversary than traffic located entirelywithin a domain.

However it must be noted that this is entirely a societal property, not at all a technical one. Two admin-istrative domains may be in cahoots with each other even though they are not believed to be. In fact, theymay be in league without realizing it—a network intruder may have surreptitiously deposited the adversaryin their territories. In short, saying that a protocol crosses administrative domains is akin to assuming apossible restriction on the spatial and temporal distribution of the adversary. As such it has “add-on value”but is not a primary technique.

1Administrative domains are called autonomous systems in the IP routing world.

Chapter 3. Essential Techniques 21

3.4 Explicit and Implicit Addressing

What makes anonymized protocols interesting and challenging is usefully maintaining the distinction be-tween addresses used for routing and those used for identification. Each node has at least one address thatidentifies it uniquely among its peers, but that is not necessarily the only way to address the node. Theaddresses that a node responds to and the addresses it uses to represent itself are important parameters of ananonymizing network.

Following Pfitzmann [67], we say that a node N ’s address A is explicit if the bits of A (or a straightfor-ward function thereof) can be used to route a message directly to N , otherwise the address A is implicit.

Suppose that Alice wishes to communicate with a node for which she has only an implicit address I .By assumption, I is insufficient to locate I’s owner, so all Alice can do is send her message to each nodethat might possibly be the owner of I . Therefore implicit addressing is typically used in conjunction withmulticasting. Note that the use of implicit addressing extends address management authority to individualnodes, who can invent as many implicit addresses for themselves as they please.

Implicit addressing with multicasting provides a straightforward technique for receiver anonymity. Areceiver N wishing to remain anonymous invents an implicit address I and publishes it in advance; later,other nodes can contact the receiver by multicasting with the name I in the message header. When N

receives the multicast, it recognizes I and processes the message. The other nodes do not recognize I andso ignore the message.

3.5 Forwarding Through a Trusted Third Party

This simple technique uses a trusted third party (TTP) willing to act as an anonymizing router for anetwork protocol. Suppose Alice wishes to transmit anonymously to Bob. To do so, she sends the messageto the TTP Trudy along with the instruction to forward it to Bob. Trudy then forwards the message to Bobwithout mentioning the name Alice. Figure 4.1 on page 36 shows an example of this technique.

Forwarding through a TTP provides receiver compatible sender anonymity with respect to a passiveexternal adversary (i.e., one close to Bob). Observers close to and including Bob only see a request comingfrom Trudy, and Alice is not mentioned. If the TTP is fast and well-placed in the network, this techniqueneed be no more costly in terms of latency and throughput than an extra routing hop. However, the techniquedoes not provide receiver anonymity: any observer can see that Bob is being addressed. In addition, the TTPtechnique can be broken by an internal adversary—one on the path between Alice and Trudy—that simplyextracts the endpoint names from the “forward to” instructions.

3.5.1 Responses in a Connection-Oriented Protocol

In a connection-oriented protocol anonymized by TTP forwarding, we must also address the issue of deliv-ering responses to the sender’s anonymized messages without betraying the sender’s identity. This elevatesthe TTP forwarding technique from a sender anonymous protocol to a client anonymous protocol. Thefollowing technique is based on reuse of a feature of the disclosed protocol (if it exists).

The multiplexing address space of a protocol determines the number of simultaneous independent con-nections possible between any two fixed nodes. If the disclosed protocol has a reasonable amount of multi-plexing address space, then we can reuse this space to distinguish between multiple anonymous connectionstraveling through a fixed forwarder T . In order to distinguish between multiple simultaneous anonymous


accesses A ! B and A0 ! B so that the responses from B can be properly delivered, the TTP T shouldconstruct at least the same number of distinct connections between T and B. Note that this techniqueprecludes connection unlinkability, since it implements a one-to-one correspondence between anonymousconnections and disclosed connections.

Some protocols already have multiplexing address space allocated to support multiple simultaneousconnections between fixed peers. TCP is one example; it uses a pair of 16-bit port numbers to disambiguateconnections. If the multiplexing space is used to route responses as suggested, then the number of possiblesimultaneous anonymous senders through the TTP is also limited by this space.

If multiplexing is used, then the TTP has to maintain state that associates the multiplexed addresses andthe corresponding senders; this either makes the connections flowing through the TTP fragile in the case ofa TTP crash, or if the TTP records its state on a persistent media, exposes the correspondences to infiltrationby adversaries able to physically raid the device.

Implicit addressing (see x3.4) can be used as an alternative if there is no straightforward way to applymultiplexing.

3.6 Cryptography in Anonymizing Protocols

Perhaps the first thing that comes to mind when contemplating sender or receiver anonymity is to simplyencrypt the message headers. While this certainly challenges the adversary, it also challenges the networkinfrastructure, leaving it unable to route the message in any sensible way. Cryptography nonetheless playsan important role in many anonymizing schemes. In conjunction with a separate routing technique, sym-metric and asymmetric cryptography can disguise and authenticate a constituent message in an anonymizingprotocol so that the adversary receives only an inscrutable and immutable object together with its size andinferred timestamp2.

3.6.1 Sender-scrutinized Receiver Anonymity

For example, refer to the TTP scheme of x3.5 and suppose Alice also shares a secret key with Trudy. Alicecan then encrypt her “forward-to-Bob” request on its way to Trudy, who then forwards Alice’s enclosedrequest to Bob in the clear as before. This gives Alice sender-scrutinized receiver anonymity against internaladversaries in addition to the sender anonymity she already had against external adversaries.

3.6.2 Private Keys as Implicit Addresses

In a system that uses implicit addressing, the private key I of an asymmetric key pair (private = I;public =

P ) can be used as an implicit address. To send a message M to the owner of the key pair, Alice multicaststhe message M encrypted under the public key P . Only the possessor of I can decrypt the message; theother nodes fail and throw away the result. This simultaneously encrypts the message M and providesreceiver anonymity to the key pair owner. If the key pair owner changes implicit addresses often, it can alsoprovide receiver unlinkability.

2An inferred timestamp is simply the value of the observer’s clock at the time of observation.


3.6.3 Inheriting Low-level Anonymity

Cryptography can also help extend low-level anonymity to high-level protocols. Suppose a low-level pro-tocol is available supporting anonymous communication between A and B. This is sufficient to establish ashared secret between A and B (using Diffie-Hellman, for instance [31, 82, 57]) that can subsequently beused to encrypt a high-level protocol between A and B in its entirety on the low-level anonymous channel.

Note that while the high-level protocol might support direct communication between A and C by auto-matic routing through B, using this encryption technique does not immediately extend the low-level ano-nymity to communication between A and C; it may only offer anonymous communication between A andB and a separate anonymous channel between B and C . Cascading anonymous channels in this fashion car-ries risks similar to those inherent in network-layer encryption, because each such junction node has directaccess to part of the protected information.

3.6.4 Surface Characteristics Persist

Message size and inferred timestamps may still be present when cryptography is used with a anonymizingrouting technique. This leads to a vulnerability that can be exploited by a distributed adversary in theforwarder tracking attack. In this attack, the adversary includes an internal node that intercepts all internalcommunication with the TTP together with an external node that intercepts all external communication withthe TTP. Suppose that Alice contacts Bob via the encrypted channel to Trudy, as before. Implementedefficiently, the size of the anonymized network message Alice! Trudy will be slightly and predictablylarger than the subsequent disclosed message Trudy! Bob. Trudy’s processing delay will be slight andpredictable as well. By correlating this information, the adversary may be able to produce compellingstatistical evidence that Alice contacted Bob. See Figure 3.2.

3.7 Resisting the Forwarder Tracking Attack

The forwarder tracking attack is based on the availability and accuracy of size and timing information. So,one way to resist the attack is to disturb the quality of that information.

3.7.1 Redefining Protocol Semantics

Both size and timing information can be hidden by sufficiently redefining the protocol semantics so that theanonymized version’s surface characteristics are independent of the disclosed version’s. However, this canbe extremely expensive and difficult.

Most client-server systems are designed on an event model where the server is idle until interruptedby a client request. Their network protocols reflect this architecture by allowing asynchronous requests toarrive from a client, and their operating systems are built to optimize the response time of asynchronousrequests. Perfectly disguising a client-server transaction in an anonymous protocol could mean turningthis architecture upside down, for instance, by having a server sometimes infer a client transaction whenobserving a sufficient lack of protocol exchanges. Such a radical transformation in semantics would requireboth intimate knowledge of the disclosed protocol’s baseline semantics and its expected use if the resultinganonymous protocol is to be at all efficient.


Alice

Trudy

Bob

Figure 3.2: The forwarder tracking attack. The adversary can observe all linksbut has no access to the encryption keys in use. Still, the adversary canconclude—or at least strongly suspect—that Alice is in communicationwith Bob on the basis of message surface characteristics alone.

Fortunately, there are generally applicable and much simpler techniques to disguise the surface char-acteristics of an anonymous protocol constructed from simple encapsulation of a disclosed protocol. Thefollowing paragraphs describe some of them.

3.7.2 Breaking the Timing Link

To break the timing correspondence between an anonymous forwarder’s inbound and outbound messages,the forwarder can use batching. A batching forwarder does not immediately decode and forward requestsfrom scrutinized nodes. It instead collects such requests and holds them for a time before acting. This locksthe adversary’s inferred timestamps for the forwarded messages to the forwarder’s release events; thesemight be triggered by wall clock time or the arrival of suitably many forwarding requests. If forwardingrequests arrive from multiple scrutinized clients during the hold time, then the correspondence between for-warding requests and forwarding actions is broken, thereby considerably weakening the statistical evidencedue to the inferred timestamps.

Note that the use of a queue with standard FIFO semantics for batching would botch the technique. Inthe forwarder tracking attack, the adversary uses only the ordering of the forwarder’s inbound and outboundmessages to reach a conclusion. Therefore, the attempt to foil the adversary by batching with a FIFOqueue would succeed only in introducing delays in the anonymous protocol. One can avoid this pitfall byextracting messages in an order independent from their insertion order; this technique is called reordering.One standard approach is to extract the messages in lexicographically sorted order. Another is to impose arandom delay on each forwarding request.

The batching technique obviously delays each message from the time it arrives at the forwarder until it


emerges. It is therefore only suited to protocols that do not suffer from such delays.

3.7.3 Breaking the Size Link

Every protocol with variable-length messages can be encapsulated in a protocol with fixed-length messagesthrough the standard networking techniques of fragmentation and padding. An anonymous protocol thatuses only fixed-length messages exposes one fewer statistic than a protocol that uses variable-length mes-sages, and hence is more resistant to forwarder tracking attacks.

Using fragmentation and padding decreases the network utilization when compared to the unfragmentedand unpadded protocol: padding occupies the network without carrying additional information, and frag-mentation repeats protocol headers in each fragment of the original message.

Another simple possibility is to apply compression before encryption to the disclosed payload portionof the anonymous message.

3.7.4 Breaking the Existence Link

But even these measures do not fully negate the forwarder tracking attack. Most protocols, in the interest ofefficiency, use the absence of a message to communicate the status “no attention required”. If an anonymousprotocol preserves these semantics from the disclosed version, then an observer can approach the identityexposure problem backwards by eliminating suspects. In Figure 3.3, the adversary observes a messagebeing forwarded from Trudy to Bob. Since Alice has sent messages to Trudy but Adam hasn’t, the adver-sary concludes that Adam is not the source of the communication. The obvious resistance to this attackis to change the semantics of the absence of a message, for instance, by having it mean “system crashed”,and using an explicit encrypted message to mean “no attention required”. Under these semantics, the num-ber of messages observed by an adversary is independent of the amount of information being transmitted.Messages encoding “no attention required” are called dummy messages.

Sending dummy messages in order to hide the lack of communication is an extremely expensive propo-sition. It consumes network bandwidth voraciously, requiring the network designer to anticipate constantheavy loads rather than infrequent peak loads. Of course there is little sense in breaking the existence linkwithout dealing with simpler threats, so it is fair to assume that the dummy messages are all encrypted.This requires nodes with nothing to report to spend CPU time encrypting and nodes that receive dummymessages to spend CPU time decrypting 3.

Suppose Alice and Adam send dummy messages to the forwarder Trudy in order to resist forwardertracking attacks against an anonymous protocol that (aside from the dummy messages) preserves the dis-closed protocol semantics by way of encapsulation. How frequently should they report that they have noth-ing to report? There are a number of possibilities.

� If Trudy does not use batching and instead forwards messages at a natural rate, then the scrutinizednodes should coordinate their contact to Trudy. Otherwise, the absence of a scrutinized node’s com-munication with Trudy quickly eliminates that node from suspicion as the source of disclosed out-bound messages. (A scrutinized node may fail to report due to excessive CPU load, crash, network

3Cryptographic hardware can help minimize this impact, but there is still a noticeable burden in sending or receiving any networkmessage.


Alice

Trudy

Bob

Adam

Figure 3.3: Eliminating suspects. Even though all messages now have identicalsurface characteristics, the absence of messages from Adam excludesit from suspicion.

failure, or other bad luck). Even noticeable differences in the relative bandwidth consumption ofscrutinized nodes can eliminate some of them from suspicion.

� If Trudy uses batching with release at preset times (e.g. every 5 minutes), then the relative frequencyof dummy messages is not as sensitive. But again, if Alice’s bandwidth consumption is much less thanTrudy’s outbound throughput on an observed connection, then Alice is probably not the connection’sendpoint. However, it is easier to match average bandwidth consumption at 5 minute intervals than itis to precisely interleave transmissions on an asynchronous network.

� If unlinkability is a design goal, then Trudy should be configured to limit her total disclosed outboundtraffic to the maximum bandwidth consumed by the slowest scrutinized node. This makes it difficultfor an adversary to conclude even how many nodes are attempting to communicate through Trudy.

3.8 Cascades of Untrusted Forwarders

So far we have considered a simple model in which a set of scrutinized nodes execute an anonymous protocolwith a fixed TTP that subsequently forwards data on behalf of the scrutinized nodes. This model requiresthat the forwarding third party be trusted, since the forwarder has access to the complete correspondencebetween the anonymous and the disclosed (compatible) protocol. This makes the forwarder a good targetfor attack and infiltration.

We can negate the value of forwarder infiltration if we can remove the forwarder from the trusted loop.Since any node can determine the immediate sender of a message it receives by simply consulting its net-


work interface, it follows that any anonymizing scheme based on single-hop forwarding implicitly trusts theforwarder to keep that information secret. Therefore, to remove the trust requirement, we must introducemultiple anonymizing hops to the picture.

Suppose that we modify the forwarding TTP model so that Alice sends her message for Bob first to Gail,who then sends it on to Trudy before it reaches Bob. With appropriate header rewriting, Trudy no longerknows that the source of the message is Alice—the message appears to come from Gail. That is fine, exceptthat Alice now must trust Gail in the same way that she used to trust Trudy; all we have done is move thetrusted party around and slowed the messages in transit.

However, now that we have multiple hops in the system, we can use cryptography to protect the for-warders from each other with a technique called layered encryption forwarding. Gail needs to know theAlice’s message has to go to Trudy, but Gail does not have to know about Bob at all. Similarly, Trudy needsto know about Bob and Gail, but Trudy does not have to know about Alice. Suppose that all of the princi-pals in this picture have well-known public keys (other than Bob, who runs only the compatible disclosedprotocol). Then Alice could format her message M for Bob as follows:4

Alice! Gail : pubGail Gail! Trudy : pubTrudy Trudy! Bob : M (3.8.1)

Here we see the outer layer specifies routing to Gail only, and the payload of the outer message is encryptedwith Gail’s public key. Gail alone can decrypt the payload, which reads

Gail! Trudy : pubTrudy Trudy! Bob : M (3.8.2)

Gail then forwards the inner payload to Trudy as suggested. Although Gail knows this message came fromAlice, she cannot read the inner payload, since it is encrypted with Trudy’s public key. Trudy alone candecrypt the remaining payload revealing

Trudy! Bob : M

that she then forwards to Bob.The layered encryption scheme does not change the amount of required trust in the system: together,

Gail and Trudy have the same information that the forwarder would have in a single-hop forwarding system.However, both nodes need to be attacked in order to recover a complete source-to-destination route. Anattacker that takes over both nodes would certainly be able to piece together the forwarding routes. Inaddition, a passive adversary observing both nodes’ network links could reassemble the forwarding routesusing the forwarder tracking attack. So, adding layers of forwarding trades increased latency and equipmentcosts for additional attack cost.

Layered encryption forwarding as described so far achieves sender anonymity for Alice, but to make thisa client anonymous protocol, we need to enable anonymous replies to Alice’s messages. There is a problemhere: since we have purposefully kept Trudy unaware of Alice, Trudy cannot construct a return path to Alicewithout assistance.

4pubX Y means encryption of Y by X’s public key.


3.8.1 Virtual Circuits

One solution is to modify the protocol to establish a virtual circuit through Alice’s chosen outbound routeidentified by randomly chosen tokens at each hop on the way. For example, suppose Alice starts as beforewith message 3.8.1. When Gail receives that message from Alice, she extracts the payload 3.8.2, choosesa random unused connection identifier idG and remembers it in a database, and forwards the followingmessage to Trudy:

Gail! Trudy : pubTrudy idG;pubTrudy Trudy! Bob : M

Then when Gail later receives a response message

Trudy! Gail : pubGail idG;M0 (3.8.3)

she looks up idG in her virtual circuit table and sees that M0 should be forwarded to Alice.As always, if an adversary can inspect all of the network links connecting forwarding nodes to the

ultimate sender and receiver, then the adversary can statistically reassemble individual communication pathsand so deduce a communication’s endpoints using a multistage passive forwarder tracking attack. Theadversary’s confidence in his conclusion improves as the number of messages on fixed route increases andas the amount of competing cross-traffic in the observed network segment decreases.

An adversary tapping Gail’s network link can deduce routes faster and with more confidence using asimple active attack. The adversary just copies the message 3.8.3 when it appears, waits for Gail’s networklink to go idle, retransmits the message to her, and observes her subsequently transmitting to Alice. Thisreplay attack can be prevented by incorporating a nonce such as a sequence number into the sealed portionof message 3.8.3, and having Gail and other forwarders ignore messages containing nonces that have alreadybeen seen.

A related vulnerability to keep in mind is that an adversary may discover a valid token such as idG anduse it with a fresh nonce to elicit a response from Gail. However, the protocol outline suggested above neverexposes tokens in the clear, so it resists this line of attack.

One weakness of virtual circuit forwarding is that it requires each forwarder to maintain state aboutits current connections. If that state information is disturbed (by a forwarder reboot, for instance), thenthe related routes are broken. If the disclosed protocol is normally forwarded by stateless nodes, then thisweakness is particularly glaring.

3.8.2 Layered Bidirectional Source Routing

For Alice to send a message to Bob using layered encryption, she has to choose the complete route throughthe forwarders in advance. This is an instance of source routing []. There is nothing stopping her fromconstructing a return route from Trudy (the last forwarder) at the same time. We illustrate with an example.Alice constructs a return path message

pubGail Gail! Alice (RP)


and includes it in her original outbound message

Alice! Gail : pubGail Gail! Trudy : pubTrudy RP;Trudy! Bob : M (3.8.4)

When Trudy receives her portion of the original outbound message

pubTrudy RP;Trudy! Bob : M

she associates RP with the connection she creates to Bob. When Bob responds to Trudy with the messageM0, she encloses it with RP on its way back through Gail:

Trudy! Gail : pubGail RP;M0

Now Gail will be able to unpack RP and see that M0 needs to be delivered to Alice. Note that Trudy is unableto read the inner portion of RP that mentions Alice’s name and Gail is unable to read the inner portion ofmessage 3.8.4 that mentions Bob’s name.

As in x3.8.1, a nonce can be used to prevent a replay attack from revealing the corresponded encoded inRP, and RP is never revealed in the clear, so only Alice and the possessor of Gail’s private key can use it.

Bidirectional source routing does not require forwarding nodes other than the anonymous/disclosingprotocol gateway to retain additional state, so a forwarder reboot does not necessarily scuttle the connectionsthat pass through it.

3.9 Resisting the Exclusion Attack

As noted in x2.4, if an adversary knows that only one message is flowing through an anonymous network,then every node observed communicating must be part of the route connecting the sought endpoints. Anadversary who arranges this situation is using an instance of the exclusion attack. In this attack, theadversary temporarily excludes competing traffic in an anonymizing network in order to trace a route.

There are at least two different ways to clear competing traffic from the network. One is to physicallyprevent competing nodes from sending; an adversary built into the networking infrastructure could accom-plish this. Of course this will only be effective in an anonymous protocol where silence is regarded as normaland acceptable.

If the above is not possible, then the adversary may try to flood the network with known, recognizablemessages so that there is just barely room for the scrutinized traffic to squeeze through. Since the floodedmessages are recognizably the adversary’s, they can be ignored as irrelevant, and the remaining messagesmust therefore be following the sought route. For instance, the sender Alice in a layered encryption systemcan recognize her message in each stage as it travels through the forwarders and loses encryption layers. Anadversary could likewise recognize flooded messages. On the other hand, the ability to recognize messagesin transit is easily revoked by adding network layer encryption between forwarding nodes.

A scheme that enforces fairness in the use of anonymizing resources may also offer some resistanceto the flooding exclusion attack. If the fairness property guarantees that each possible sender has equalaccess to the protocol gateways, then an adversary intent on flooding would have to make the flood appearto come from many different scrutinized nodes, which in turn might imply that the adversary is alreadywidely distributed (so a forwarder tracking attack might be applicable) or that the adversary has infiltratedmany of the forwarding nodes.


3.10 Mixes

In [17] Chaum introduced the concept of a Mix and in so doing inaugurated the study of anonymizingnetworks. In the language we have introduced so far, a mix is a cascade of untrusted forwarders usinglayered encryption with nonces, bidirectional source routing, fragmentation and padding, and batching withreordering at each forwarding node. Chaum specified the generation of receipts based on digital signaturesthat ensure scrutinized nodes of the proper processing of their messages by the anonymizing network as well;he also considered the possibility of using dummy messages to hide the lack of communication. Severaldeployed and proposed techniques trace their ancestry either directly or indirectly to the Mix technique andare described in Chapter 4.

3.11 Superposed Sending

Any anonymizing network able to use dummy messages to hide the existence or lack of communication(see x3.7.4) while still performing acceptably is a candidate for the use of the superposed sending method.Superposed sending, called the Dining Cryptographer (DC) technique in the original paper by Chaum [18], isa class of scalable multiparty cryptographic transformations providing sender anonymity. A single networkcommunication round produces a single message on behalf of the entire collective of scrutinized nodesby combining key material from each of them. The individual network messages are encrypted in such away as to be undecipherable not only to an adversary that observes every network link but also to certaincooperating subsets (specified as a protocol parameter) of malicious anonymizing nodes.

In other words: within a single round, no algorithmic entity can predict any fair participant node’soutput5 given that node’s input and the internal state of each member node of a fixed cooperating subset.Only at the end of a round—after all nodes have participated—is the combined message intelligible to anyparty.

Let us illustrate with a simplified instance of superposed sending based on multiple applications of one-time pads. We assume a network of five anonymizing nodes labeled 1; : : : 5 and arbitrarily designate node1 to be the anonymous/disclosed protocol gateway as well. Let us also assume that the standard messagelength in the disclosed protocol is 12; 000 bits (the size of a maximum-length Ethernet frame). Each of the�52

�pairs of nodes in the anonymizing network randomly chooses and shares a 12; 000-bit one-time pad in

advance. For a pair i; j of nodes, let Pfi;jg(= Pfj;ig) its one-time pad.Now assume that node 3 wishes to transmit a message M3 across the gateway and no other node desires

to transmit at that time. We represent this by setting Mi = 0 for each i 6= 3. To make their desires known,each node i 2 f1 : : : 5g computes and transmits the encrypted message

Ei = i! 1 :Mi

Lj 6=i Pfi;jg

where � denotes the bitwise exclusive-or operation and each Mi has been padded to 12,000 bits. (Node 1need not actually transmit the message to itself.)

Collecting all of the intermediate messages Ei, node 1 exclusive-ors them all together. In the result,each Pfi;jg cancels out its corresponding Pfj;ig and so node 1 is left with the sum

M1 �M2 �M3 �M4 �M5:

5Excluding that node itself, of course.


Since only M3 is nonzero, this sum is equal to M3. Node 1 can then gateway M3 to the disclosed protocol.Some observations:

� This technique reliably delivers the exclusive-or sum of the Mi to node 1, but it does not directlyindicate how to disassemble the sum into the original parts. Our example was contrived because onlynode 3 tried to transmit. This is called the collision resolution problem in superposed sending.

� Any four of the nodes can conspire to determine if the remaining node sent a message. (If none of thefour sent it, then the fifth must have.)

� Less obviously, no group of three or fewer nodes can successfully determine which of the remainingnodes sent a message. Suppose nodes 1, 4, and 5 want to know whether node 2 was silent and node 3transmitted during a round or vice-versa. Cooperating fully, nodes 1,4, and 5 know all of the one-timepads except Pf2;3g(= Pf3;2g). But both M2 and M3 are combined with that one-time pad before beingtransmitted, and a single unknown one-time pad is sufficient to make a transmission illegible to anadversary [85]. The adversary can determine M2 �M3, but that does not help determine which nodeis responsible for which of the bits.

� Network infrastructure attacks (i.e., attacks by network entities having full access to the wires but nospecial access to any of the cryptographic key material) can be no more successful than an attack bynode 1, which receives and processes all of the intermediate anonymized messages. By the previousobservation, a network infrastructure attack can be considered—at worst—an attack by a single node.

Therefore this particular protocol is resistant to attack by any subset of 3 or fewer nodes. In general, asuperposed sending network of n nodes can be made resistant to attack by any subset of n � 2 nodes.If a designer is willing to risk successful attacks by fewer nodes, then the protocol’s performance can beimproved by generating fewer one-time pads.

Since the designated gateway is unable to even read a communication round’s message until the one-time pads are combined to cancel each other out, the gateway has to wait for the slowest scrutinized nodeto submit its contribution before emitting the disclosed protocol version of the round’s output. Superposedsending is therefore most appropriate when the scrutinized nodes are able to compute and communicate atroughly equal performance levels.

As described, superposed sending through a gateway only supports sender anonymity (although [18] and[67] describe how to achieve receiver anonymity and unlinkability when both endpoints are able to directlyrun the anonymous protocol). In order to make it a client anonymous protocol, it must be combined with areceiver anonymity technique such as implicit addressing or virtual circuits so that responses do not betraya client’s identity.

In formulations using random one-time pads, superposed sending has been proven [18, 96] to provideinformation-theoretic secrecy of message contents (and therefore their senders) in the sense of Shannon [85].In contrast, the other techniques in this chapter rely on ad hoc measures to disrupt known attack methods orat best rely on the complexity-theoretic secrecy of public-key cryptosystems.

3.12 Summary

In this section we have examined the essential techniques used when deriving an anonymous protocol froma disclosed one. The techniques revolve around the use of the necessary forwarding nodes and have impact


on the type of anonymity and/or unlinkability achieved, fault tolerance, vulnerability to attack, and perfor-mance; there is no one solution for all situations. Table 3.1 coarsely summarizes the techniques and theireffects.


Send

erA

non.

Clie

ntA

non.

Send

er-v

uln.

Rec

eive

rA

non.

Rec

eive

rA

non.

Serv

erA

non.

Rec

eive

r-vu

ln.

Send

erA

non.

Stat

eles

sFo

rwar

der

Faul

tTol

eran

ce

Unt

rust

edFo

rwar

der

Low

Lat

ency

Hig

hU

tiliz

atio

n

Randomized Routing + + + + + + + - - -

Crossing Administrative Domains + + + + + + - - -

Implicit Address for Vuln. Sender +u +u +u + +

Implicit Address for Vuln.Receiver

+u +u +u + + -

Virtual Circuits -u + -u + -u + -u + -u + -u - - - + +

Scrutinized Endpoint ChoosesRoute

+u +u +u +u +u +u + + +

TTP Forwarding + + + + + + - + +

Untrusted Cascades + + + + + + + -

Batching + + + + + + - - +

Reordering + + + + + + - -

Uniform Sizing + + + + + + - -

Dummy Messages + + + + + + - - -

Fairness in Gateway Access + + + + + +

Superposed Sending + + + - - + - -

‘+’ means the technique benefits the property.‘-’ means the technique hinders the property.‘+u’ means ‘+’ and the technique benefits the unlinkability form of the anonymity property.‘-u’ means the technique hinders the unlinkability form of the anonymity property.‘ ’ means the technique has no intrinsic significant impact on the property.

Table 3.1: Summary of essential techniques and their impact on properties. Columns are labeled interms of desirable properties, e.g., using an “implicit address for a scrutinized sender”helps a protocol’s ability to “distrust forwarders.”

Chapter 4

Prior Work

In the previous chapter, we described techniques that can be combined to achieve various types of anonymity.In this chapter we survey the combinations have been attempted. Most of the techniques have appeared ineither production or prototype anonymizers; only the costliest techniques (dummy messages, superposedsending) are not part of the current literature. At the end of this chapter, we summarize our view of the priorwork in two tables.

The schemes offering a high degree of anonymity are generally Mix-derived. However, two low-securitysimple rewriting schemes have dominated the Internet anonymity field in practice: the Penet mail rewriterand the (Web) Anonymizer.

4.1 Anon.penet.fi

The Internet host anon.penet.fi (abbreviated here as Penet), maintained by Helsingius, ran as a trustedthird party cleartext SMTP forwarding agent from 1993 until it closed in 1996. It offered client and serveranonymity for recognized users by maintaining a private database associating real and assigned anony-mous email addresses and rewriting addresses appropriately when mail was directed through it. Upon usingthe service for the first time, the forwarder generated a permanent anonymous email address of the [email protected]. The extremely long-term virtual circuit named by such an address couldthen be used by another party to contact the original sender. Figure 4.1 depicts Penet’s operation.

Subpoenas demanding the secret mapping for specific email addresses were an obviously major factorin the service’s demise; see [42, 61] for details. At its peak, the service claimed a mapping for 500,000 usersand was indisputably the most important service of its kind.

Since Penet used only a single, fixed forwarder and no encryption, the simplest conceivable forwardertracking attack would have defeated it. It is a testament to the effectiveness of crossing administrativedomains that no claims of defeat on technical grounds were made during its three years of operation.

4.2 Cypherpunk Remailers

The Cypherpunk remailers forward email via SMTP as a cascade of store-and-forward header rewritingnodes. Because of development history, encryption is not a mandatory part of the protocols. However, mostof the remailers do support PGP encryption [99] for protection of message headers and contents and thus

36 Chapter 4. Prior Work

anon.penet.fi

alice.net

From

: Alic

e@al

ice.n

et

To: B

ob@

aol.c

om

From: an1057@

anon.penet.fi

To: [email protected]

aol.com

[email protected] = [email protected] = an3424etc.

Figure 4.1: Operation of anon.penet.fi. The email recipient does not comeinto direct contact with alice.net, and the email headers have beenrewritten to omit that name as well. The automatically generated“from” address can be used to return a reply to Alice.

protection against attack by rogue forwarders. Protocol-specific control information is encoded in SMTPheaders or the message body itself.

The installed forwarders are believed to not record traffic, making after-the-fact analysis extremely dif-ficult. Without the persistent database, no anonymous ID is available to enable return mail. Since there isno single point of presence, and since the individual remailers are very easy to set up, legal action or net-work attacks are unlikely to threaten the system as a whole. While court action or service provider protestcould easily shut down a remailer, there is no database to raid. Meanwhile, another remailer would appearelsewhere.

Cypherpunk remailers are widely available; see [33]. Figure 4.3 shows the traffic flow in a Cypherpunkremailer chain, and Figure 4.2 shows a recent snapshot of “well-known” remailer status.

4.3 The Mixmaster and Babel Remailers

The Mixmaster for email by Cottrell1 [21] is probably the most perfectly suited implementation of Mixtechnology to date. It includes facets of the Mix design such as replay detection, uniform sizing, batching,and reordering, all of which contribute to the untraceability of messages. The technology is almost as easyto deploy as Cypherpunk remailers, the major difference being that one needs to use a special client forsending mail with the Mixmaster. The Mixmaster is currently available within the U.S. at [22].

1Also President of Anonymizer, Inc. (See x4.5).

Chapter 4. Prior Work 37

Last update: Mon 16 Nov 98 17:56:27 ESTremailer email address history latency uptime-----------------------------------------------------------------------hyper [email protected] --+**+**-*** 16:13 100.00%nym [email protected] ##++##+#*#*# 3:16 100.00%dongco [email protected] *-+**+#*-### 8:42 99.99%replay [email protected] ***-+*-*+*** 8:18 99.98%redneck [email protected] ############ :16 99.98%nowhere [email protected] ***+*-+**+** 1:38:43 99.97%base [email protected] ------------ 3:42:21 99.92%squirrel [email protected] ----------- 2:13:28 99.88%cracker [email protected] ----- ----+- 1:13:08 99.84%mix [email protected] +--++ ---** 5:05:10 98.24%hr13 [email protected] - - -- 1:38:18 93.90%lo14 [email protected] --*** *-*+ 18:37 87.49%tea [email protected] --- - - 13:28:05 78.86%bong [email protected] 6:42:25 0.00%

History key

# response in less than 5 minutes.* response in less than 1 hour.+ response in less than 4 hours.- response in less than 24 hours.. response in less than 2 days._ response in more than 2 days.

Figure 4.2: A recent list of Cypherpunk remailers. Obtained fromhttp://www.efga.org/�rlist [33]


redneck.efga.org

alice.net

pubredneck Redneck -> Base: pubbase Base -> [email protected]: Message body

base.xs4all.nl

Base -> [email protected]: Message bodypubbase

aol.com

Message Body

From: [email protected]: [email protected]



Figure 4.3: This illustration of Cypherpunk remailer operation shows a straightfor-ward application of the cascaded untrusted forwarder technique (seex3.8). The Mixmaster also follows this pattern but uses batching withreordering and uniform sizing to further disguise flows.


Tsudik and Gulcu at IBM Zurich developed a similar remailer called Babel, described in detail in [41],that can be considered a proprietary research-world version of the Mixmaster. Babel includes a mechanismfor bidirectional source routing to enable anonymous reply to anonymous messages. The Babel implemen-tation is not currently available to the public.

Generally speaking, the Mix design works well with SMTP because email tends to be insensitive tonetwork latency, and latency is a direct consequence of batching and reordering. Furthermore, SMTP allowsnew headers to be introduced at will, and its extremely free-form body can also be used to carry anonymizingprotocol instructions and parameters.

4.4 Nym.alias.net

Although the Mixmaster and Cypherpunk remailers do not provide for return mail by virtual circuits, thebidirectional source routing option remains viable. When preparing an outbound anonymous email, a usercan select an encrypted layered return path and paste it onto the email together with instructions for therecipient on how to use it. This in essence extends server anonymity to email recipients using the aboveremailers; the mail recipient chooses the forwarding path and gives it to the sender without revealing the tailend of the path.

An alternative to manual intervention is to use the nym.alias.net service (abbreviated here as“Nym”) to automate the process. Nym operates like Penet in principle, except that instead of associat-ing anonymous identifiers with “real” email addresses, it associates them with encrypted forwarding paths.Nym commands, such as requests to establish an anonymous identity or send a piece of outbound email, areissued in the body of email messages. This allows users to push a (possibly very thick) layer of Cypher-punk/Mixmaster remailers between Nym and themselves for all operations.

For example, Alice could use Mixmaster to send a message creating the (poorly chosen) email [email protected] for herself. To send mail “as” Aliceanon, she creates an email messageencapsulating her outbound email and sends it to [email protected]. Nym, recognizing that theoutbound send request was signed by a holder of Aliceanon’s PGP key, sends the message on her behalf.Mail that is sent to [email protected] is automatically routed by the Nym service throughthe encrypted forwarding path that associated with Aliceanon’s account when she established it, presumablyback to Alice.

Nym does a much better job of protecting its users than Penet did. If its database is raided, all that canbe obtained is the binding between anonymous email addresses and encrypted forwarding chain. In order tounravel the true owner of an anonymous identifier, all of the applied encryption would have to be broken bytechnical or legal means—not a likely event.

However, a replay attack on an obtained forwarding chain can confirm or deny a suspicion. If Evesuspects that Aliceanon is really Alice’s account, and Eve can observe Alice’s network interface (perhapsby court order), then Eve can just send mail to [email protected] and wait for it to arrive ornot arrive at Alice’s computer. The results would be extremely persuasive.

4.5 The (Web) Anonymizer

The Anonymizer http://www.anonymizer.com, provided by Anonymizer, Inc., can be thought of asa Penet-style remailer for HTTP [7, 36] traffic. Since HTTP transactions are completely self-contained, no


long-term mapping for return addresses is required. The Anonymizer just removes revealing HTTP headersand relays the results. As with Cypherpunk remailers, subpoenas are intrinsically insufficient to recover theidentity of a past user: once an HTTP transaction completes (typically in seconds), the Anonymizer canforget the mapping used, and so be truly unable to comply with such demands.

However, Anonymizer, Inc. has made some interesting policy decisions regarding the use of their ser-vice, most of which seem fully informed by the legal reality of doing business in California and the UnitedStates. In the Terms and Conditions as of September 24, 1997 [2], they write

8.3 Usage logs are usually kept for fifteen (15) days for maintenance purposes, monitoringSpamming and monitoring abuses of netiquette. Any relevant portion(s) of such logs may bekept for as long as needed to stop the abuses.

In addition, their FAQ clearly states their intention to use these logs in order to cooperate with appropriateauthorities. So the service closely parallels Penet in terms of confidentiality even though the same technicalguarantees of untraceability offered by Cypherpunk remailers are possible. Unlike Penet, Anonymizer, Inc.does not intend to shut down if it is merely required to reveal a user’s identity.

4.6 TAZ Servers and the Rewebber Network

In [38], Goldberg and Wagner describe a Cypherpunk-style cascade of “remailers” for Web traffic called a“Rewebber” network. Their Rewebber system is most directly aimed at hiding the true publisher (server)of a web page as it continues to interactively serve the documents. The technique they use is a cascade oflayered encrypted forwarders between the client and the server. The content publisher somehow publishesan entire layered encrypted path encoded as a URL in advance, say, by using Mixmaster remailers to postanonymously to a newsgroup. Then any client can use the published URL to contact the publisher’s site.

Each level of the cascade implements its forwarding hop using standard HTTP proxying techniques. Inother words, most proxies will simultaneously maintain two HTTP connections: one with the predecessor(towards the client), and one with the successor (towards the server) in the chain. Not only does this provideserver anonymity, but the cascaded forwarder technique also potentially provides client anonymity withrespect to adversaries located close to the server. There is no batching or other attempt to break timingrelationships because of the interactive nature of Web transactions.

Although the public-key cryptosystem protects encoded URLs from off-line attacks, the Rewebber sys-tem appears to be susceptible to the suspicion-confirming replay attack suggested in x4.4.

The authors also introduce TAZ [11] servers, essentially nym.alias.net-like nodes that turn mean-ingful names into Rewebber layered paths. This shields users from having to deal directly with the extremelycomplex path URLs, such as the one pictured in Figure 4.4.

4.7 JANUS

The JANUS system [30], developed by Demuth and Riecke at the FernUniversitat Hagen, shares manycharacteristics of the Rewebber system even though the two projects were independent. JANUS also usesa layered encrypted, untrusted, non-batching cascade of HTTP proxies to achieve server anonymity, simul-taneously achieving a measure of client anonymity. JANUS paths are encoded in URLs, and JANUS issusceptible to replay attacks.


http://rewebber.com/!RjVi0rawjGRT50ECKo_UBa7Qv3FJlRbyej_Wh10g_9vpPAyeHmrYE1QL1H2ifNh2Ma4UYt3IaqeQRXXd7oxEvwR8wJ3cnrNbPF6rc1Uzr6mxIWUtlgW0uRlL0bGkAv3fX8WEcBd1JPWGT8VoY0F1jxgPL7OvuV0xtbMPsRbQg0iY=RKLBaUYedsCn0N-UQ0m5JWTE1nuoh_J5J_yg1CfkaN9jSGkdf51-gdj3RN4XHf_YVyxfupgc8VPsSyFdEeR0dj9kMHuPvLivE_awqAwU_3Af8mc44QBN0fMVJjpeyHSa79KdTQ5EGlPzLK7upFXtUFcNLSD7YLSc1gKI3X8nk15s=RXbQaqm0Ax4VhKPwkLVK_MMJaz9wchn_pI48xhTzgndt5Hk09VToLyz7EF4wGH3XKPD7YbKVyiDZytva_sUBcdqpmPXTzApYLBnl4nDOy1o1Pu1Rky8CxRfnC9BvQqof853n99vkuGKP9K4p3H7pl6i8DOat-NrOIndpz5xgwZKc=/

Figure 4.4: An encrypted layered path encoded as a URL. This example is from theRewebber paper [38], but JANUS paths take roughly the same form.

JANUS encrypts only the addresses but not the contents of pages it proxies, but the authors attributethis as a consequence of the current prototype nature of JANUS [30]. But JANUS does parse the HTML itproxies, carefully rewriting embedded URLs as JANUS paths so that they volunteer no cleartext hints as tothe real server address.

One could argue that a publisher should be smart enough to remove identifying information such asURLs from the information they anonymously publish. In addition, it is not uncommon for a single HTMLpage to contain hundreds of links such as embedded images or bookmarks; the computational expense ofcomputing multiple layered public-key encryptions for each of these at every such page access is somewhatalarming. However, the JANUS approach does allow anonymous references to sites to be generated withoutthe advance cooperation of those sites.

The JANUS service can be used at [29]. Figure 4.5 illustrates its behavior.

4.8 A Pedagogical Superposed Sending Network

The first implementation of the Superposed Sending technique is described in Bottger’s 1990 Diplomarbeit[15]. The author first constructed a networking platform called PASNetz built on AppleTalk at the KarlsruheInstitut fur Rechnerentwurf und Fehlertoleranz. Using this network, the author developed a version of asuperposed sending protocol intended to be used to demonstrate the technique to students. The Diplomarbeitcontains design information, source code (in Pascal), exercises, and performance information.

The Macintosh platform available for this project was not very well-suited to network development.Since the OS was fundamentally single-threaded and there was no buffering network interface on the systemnodes, a message sent to a node that was not spin-waiting on the network would simply be dropped and along timeout would ensue. Similarly, collisions on the (bus) network were not sensed, so these would alsocause long timeouts. Bottger must have exercised admirable patience in his development efforts.

The age of the implementation and unfriendly platform environment fully account for the lacklustergateway throughput of � 8 mbit/sec for 5 nodes with � 1500 bytes/packet on a bus network capable of


janus.vernuni-hagen.de

alice.net

http://ja

nus.fernuni-h

agen.de/janus_encrypted/MTBFmQyD0DyEJMpSV5zst...

www.fbi.gov

Connection fromjanus.fernuni-

hagen.de

Who am I reallyconnecting to?

Figure 4.5: Operation of JANUS and Rewebber networks. Both server anonymityschemes suggest the use of a chain of forwarders, but at the presenttime only one server appears to be in operation.

carrying up to 230 kbit/sec. However, the reported trends (gateway throughput vs. number of nodes) isconsistent with our findings in x10.9.1.

4.9 A Pedagogical Mix Network

An implementation of the Mix design was first described in Ort’s 1991 Diplomarbeit [66]. This paperdescribes a Mix network built on Bottger’s PASNetz and a series of student exercises. At the time, the Mac-intosh OS did not support multiprogramming, so nodes in the experiment were either purely Mix forwardersor specialized application endpoints. Like the Superposed Sending project in the same lab, the system wasconstructed primarily for educational purposes.

4.10 Onion Routing

Reed, Syverson, and Goldschlag at the U.S. Naval Research Laboratory developed an implementation basedon cascades of untrusted forwarders called Onion Routing [40, 93]. Their implementation defines protocolsfor bidirectional streams, currently rlogin, HTTP, and SMTP (email). The forwarding nodes called onionrouters maintain permanent encrypted TCP connections to each other, and the Mix structure is built on topof that virtual network. Client applications encounter the anonymous network through the application proxymodule of a fixed and trusted onion router. The proxy then operates on the client’s behalf.

When opening an anonymous connection stream (e.g., an HTTP GET from a Web browser), a proxychooses a random path through the Mix network. As this path is constructed, a randomly-generated key of a


1

2 3

6alice.net

Connectionfrom OnionRouter 5

54

Figure 4.6: Onion Routing network operation. The light lines show long-termsymmetric-key encrypted connections. To build a new connection, Al-ice’s proxy chooses a random path through the network defined by thelong-term connections that ends at the destination; this path is shownin dark lines. The connection then flows along that path, encrypted bya different symmetric key at each hop.

symmetric cryptosystem is deposited at each incident router. The resulting link-encrypted virtual connectionis then used to carry application data in both directions. See Figure 4.6.

Like the other interactive Mix-based systems, Onion Routing does not use batching at the Mix nodesto break timing links; it remains vulnerable to forwarder tracking attacks. See [75] for notes on a trafficanalysis project that discovered unwelcome correlations in an experimental Onion Router network. OnionRouting also appears susceptible to the suspicion-confirming replay attack. However, since layered paths arechosen automatically and used only for short-term connections, the use of nonces would effectively combatthis threat.

Several experimental onion networks have been established, and a next-generation system is in progress.See [76] for information on the current availability of Onion Routing.

4.11 Crowds

Reiter and Rubin of AT&T Research developed a system called Crowds [80] with the same purview asOnion Routing but only loosely related to the Mix design. As in Onion Routing, anonymous traffic visitsa secret sequence of forwarding nodes (called “jondos”) represented by a sequence of path identifiers andprotected by symmetric keys, and client applications use proxies to access the anonymous network. UnlikeOnion Routing, an originating proxy does not choose its own path. Instead, at the Crowds network boottime, each forwarding node creates and sends a special path construction packet on a random walk throughthe network. On receiving such a packet, a node flips a coin to decide whether to lengthen the path with a


1

2 3

6alice.net

Connectionfrom Crowds

Node 5

54

Figure 4.7: Crowds network operation. To enter the network, Alice’s proxy ini-tiates a random walk through the network. Each of Alice’s connec-tions follows this same path; only the last hop to her desired destina-tion changes between connections. Encryption with a single shared keyalong a path protects the messages in transit.

randomly chosen successor node or to conclude the path at the current node. (The expectation and varianceof the path length can thus be controlled by adjusting the coin’s bias.)

After this phase of random walks, the Crowds network state contains a set of secret paths. Each Crowdsnode has exactly one secret path beginning at that node and ending at another node (possibly the same one).The chosen secret path is used for all subsequent traffic originating at that Crowds node. That is, whena client wishes to perform an anonymous transaction (such as a Web browser issuing an HTTP GET), itcontacts the proxy side of its trusted Crowds node, which then routes the client’s stream along the singleCrowds path originating at that node. Each node along such a path shares a single symmetric key that isused to protect data in transit. See Figure 4.7.

Crowds has some advantages over Onion Routing. First, it requires much less encryption: a single sym-metric key is shared by every node on a path and used to protect the traffic, and no asymmetric cryptographyis required at all. Second, the designers have built a mechanism that allows both users and nodes to join andexit the network easily. In contrast, most of the other anonymous network implementations (including ourown) do not easily adapt to topology changes. And third, the ability of each node to see the final destinationof a packet allows it to choose an alternate path in case of failure. This is not automatic in a classical Mixnetwork because each Mix node can only see its immediate neighbors on a path2. However, the advertise-ment of final destinations to every encountered node may also be seen as a disadvantage, since it stands atodds with receiver anonymity.

Crowds has an application-specific timing vulnerability that its designers have worked hard to minimize.

2Techniques to address this deficiency are described in [68, 67], but we are aware of no layered encryption implementations thatuse them.


The problem is that the untraceability of a route follows from each node not knowing its position on a route,i.e., whether it is the second or third hop or greater. If an infiltrated jondo knew that it was the second hop ona route, then its predecessor (which it can see) would be entry point of the supposedly anonymous sender.

The attack follows from the propensity of HTTP transactions to occur in bursts. The first transactiontypically fetches HTML code. A Web browser quickly parses it and then issues more HTTP transactions tofetch embedded images. An infiltrated Crowds node armed with knowledge about other nodes’ speeds couldtherefore draw conclusions about its path position based on the delay between linked HTTP transactions.Crowds minimizes the attack by anticipating this behavior and delivering embedded images along with theoriginal document, so there is no burst of requests to time.

Crowds does not use dummy traffic, so it too is susceptible to forwarder tracking attacks. Crowdscurrently anonymizes HTTP only. See [23] for more information.

4.12 The Non-Disclosure Method

The Non-Disclosure Method (NDM) of Fasbender, Kesdogan, and Kubitz [35] hides the source and desti-nation addresses of an IP packet from every node except the packet’s destination. Their primary applicationis in mobile IP, where a mobile computer’s current location (i.e., its transient foreign agent or “care-of”IP address) should not be made public. NDM ensures this privacy by tunneling packets from the mobilestation’s permanent home network through a layered encryption network to the foreign agent responsiblefor final LAN delivery. Figure 4.8 illustrates this flow.

As in Onion Routing and Crowds, packets are neither delayed nor reordered at the internal forwardingnodes. Since the packet’s destination node (e.g., the mobile IP station) receives the packet’s source address,it knows where to direct its reply—no reply onions are required. This allows every packet to be routed usingan independently chosen Mix path. In addition, each packet creates a transient virtual circuit on the wayto its destination. If the destination turns out to be unreachable, the intervening nodes can generate andtransmit the appropriate ICMP [72] error packet to the source.

In [34], the same authors compute a probability distribution predicting the latency added by NDM usingequipment assumptions accurate in 1995. For a route through three forwarding nodes equipped with RSAhardware, they estimate an additional delay of less than 100 milliseconds, which suggests that the systemwould be practical for interactive use.

4.13 The Lucent Personal Web Assistant

The Lucent Personal Web Assistant due to Bleichenbacher, Gabber, Biggons, Matias, and Mayer [12, 13, 37](abbreviated LPWA) provides client anonymity and unlinkability above all, not between packets or HTTPtransactions, but between different sites. The idea is to prevent several sites providing Web service fromcoordinating their information about a user based on the IP address or information that the user volunteers,such as a username and password. LPWA runs as a trusted proxy. When LPWA receives a client HTTPrequest, LPWA removes information about the client and forwards the request to the server, essentially likethe Web Anonymizer.

However, LPWA also parses the user’s outbound data, performing keyword substitution on explicitidentity fields such as nu denoting “user” and np denoting “password”. These fields are replaced with apseudonym: a cryptographic hash of the user’s real identity together with the server name. Therefore, by


FBI LAN

evil.com

alice.net alice.netbase station

Connectionwith

alice.net

Figure 4.8: Non-Disclosure Method operation. The host evil.com believes itis communicating with alice.net, but it is really only in contactwith Alice’s base station. The base station relays each packet for Alicethrough an independently selected encrypted cascade of forwarders toAlice’s current network location. Alice likewise chooses paths for herpackets to evil.com. The light transmissions are ordinary IP.


routinely using nu in the username field and running through the LPWA proxy, a different pseudonymoususername is presented at each accessed web site. Other fields such as np for password and n@ for emailaddress are similarly processed.

This can all happen with pure hash functions; however, LPWA also records the association between realidentity and pseudonyms so that email sent to a pseudonym can be redirected to the associated real identity.Forwarder tracking and replay attacks are possible here. See [54] for more information.

4.14 PipeNet

Although no PipeNet has been constructed, its design by Dai [24] is similar to those of Crowds, OnionRouting, and this work3. PipeNet can be roughly envisioned as an Onion Routing network that also carriesdummy traffic. Each node that carries a connection forces all of that node’s active connections to run at thesame rate. PipeNet only claims to provide client anonymity.

Recently, Dai updated the proposed protocol with anti-replay protection. See the URL in [24] for details.

4.15 Zero-Knowledge Systems

Zero-Knowledge Systems Inc. is developing a commercial version composing many of the techniquesmentioned here. They envision the establishment of a number of “freedom servers”, apparently Mix nodesin separate administrative domains, for interactive Internet connections and email forwarding, perhaps alongPipeNet lines. In addition, they mention “multiple digital identities” in a manner reminiscent of LPWA’spseudonyms. See the firm’s white paper or Web site [100] for more information.

4.16 Summary

In this section we described deployed anonymizing systems in the language of Chapters 2 and 3. Emailanonymizers and the Web Anonymizer seem to be the most successful so far; the former probably becausesupporting software is portable, widely available, and extremely effective, and the latter probably because itis so easy to use.

Table 4.1 summarizes the properties of each of the systems. The table does not show how strong theoffered properties are. For instance, it shows the Web Anonymizer and Onion Routing as both offeringsender anonymity. In their own way they each do, but Onion Routing is resistant to much more powerfuladversaries than the Anonymizer. Table 4.2 shows which essential techniques from Chapter 3 are employedby each system.

3Our system design was largely complete by the time we learned of the PipeNet discussions. This seems to also be true ofCrowds and Onion Routing, as their primary expository papers do not cite or refer to PipeNet.


Send

erA

non.

Clie

ntA

non.

Send

er-v

uln.

Rec

eive

rA

non.

Rec

eive

rA

non.

Serv

erA

non.

Rec

eive

r-vu

ln.S

ende

rA

non.

Stat

eles

s

Faul

tTol

eran

ceM

easu

res

Unt

rust

edFo

rwar

der

Inte

ract

ive

Use

Forw

arde

rE

nter

/Exi

tMec

hani

sm

App

licat

ion-

orie

nted

Serv

ice

Cur

rent

lyA

vaila

ble

Anon.penet.fip p p

Cypherpunk Remailersp p p p p p p p p

Mixmasterp p p p p p p p

Babelp p p p p p p

Nym.alias.netp p p p p p p p

Web Anonymizerp p p p p p

TAZ/Rewebberp p p p p p p p p p p

JANUSp p p p p p p p p p p p

Karlsruhe SS-netp p p p p p

Karlsruhe Mix-netp p p p p

Onion Routingp p p p p p p p

Crowdsp p p p p p p p p p

Lucent Personal Web Assistantp p p p p p

Non-Disclosure Methoda p p p p p p p p

PipeNetp p p p p p

This Workp p p p p p p

Table 4.1: Features of related projects.

aAnonoymity in NDM is with respect to the care-of address.


Ran

dom

ized

Rou

ting

Cro

ssA

dmin

.Dom

ain

Impl

icit

Add

ress

ing

Vul

n.E

ndpo

intC

hoos

esR

oute

Lay

ered

Enc

rypt

ion

Forw

ardi

ng

-w

ithV

irtu

alC

ircu

its

-w

ithB

idir.

Sour

ceR

outin

g

Bat

chin

g

Reo

rder

ing

Uni

form

Sizi

ng

Dum

my

Mes

sage

s

Fair

Acc

ess

toG

atew

ay

Supe

rpos

edSe

ndin

g

Ant

i-R

epla

yTe

chni

ques

Anon.penet.fip

Cypherpunk Remailersa p p p p p p p

Mixmasterp p p p p p p p

Babelp p p p p p p p p

Nym.alias.netp p p p p

Web Anonymizerp

TAZ/Rewebberp p p p p

JANUSp p p p p

Karlsruhe SS (isolated net)p

Karlsruhe Mix (isolated net)p

Onion Routingp p p p p p

Crowdsp p

Lucent Personal Web Assistantp

Non-Disclosure Methodp p p p p p

PipeNetp p p p p p p p p p p

This Workp p p p p p

Table 4.2: Techniques employed by related projects.

aThere are many different Cypherpunk remailer implementations. We list the techniques used by the most sophisticated of these.

Chapter 5

Local Anonymity

In this chapter we define and discuss local anonymity, a new form of anonymity complementing the pri-vacy offered by the currently deployed schemes described in Chapter 4. We confine ourselves to generalproperties here; Part II of this text describes specific realizations.

5.1 Motivation and Definition

The deployed anonymity schemes described in Chapter 4 share certain characteristics:

� They are all third-party or dispersed schemes. In other words, the designs require the constructionand support of an independent infrastructure to provide an anonymous rewriting service. This in turnimplies:

� The traffic that is anonymized is long-distance. It would be senseless to needlessly subject localtraffic to the congestion and failure modes of a remote network. Therefore the model is that of a clientroaming broadly (or a server listening to a wide variety of clients).

� As a consequence, the protocols present adversaries with a very large initial search space: with internetconnectivity, the endpoint could be any internet node.

� Most of the multi-hop forwarding schemes uses randomized routing in order to minimize contact withan immobile adversary.

� While the schemes defend against a variety of attacks, none of the interactive ones offer much resis-tance to coordinated adversaries using the forwarder tracking attack. This makes sense, because thetechniques for resisting it all have a profoundly negative impact on latency (see Table 3.1).

� In order to compensate for this, the scheme’s forwarding nodes are assumed to be placed in separateadministrative domains.

Let us name an alternative. A local anonymizing network is an anonymizing network that does notrely on another administrative domain in order to maintain anonymity for its scrutinized nodes. Conversely,a dispersed anonymizing network does rely on rewriting in another domain to maintain anonymity. Weconsider all of the anonymizers of Chapter 4 (all currently existing anonymizers) to be dispersed ones.

52 Chapter 5. Local Anonymity

As a rule of thumb, an anonymizing network is dispersed if its anonymizing nodes have no stronginterdependencies other than those following directly from the anonymizing function. We expect that atypical dispersed anonymizing network will have significantly more nodes than a typical local anonymizingnetwork because while there are many, many administrative domains, only a small number of those domainsare large.

There are some purely logistical issues influencing the choice of an anonymizing network type. Onemight easily turn to a dispersed anonymizing service because of reluctance to support yet another networkprotocol on local equipment. In addition, one might be able to lease the use of an anonymizing servicethat provides better performance than could be obtained with the same amount of money spent on localequipment and protocol deployment.

But from a security point of view, the main reason to prefer a dispersed anonymizing network is that theadversary’s search space is directly related to the number of endpoints using the anonymized protocol. Byrelying on a common forwarder, the scrutinized parties actually improve each others’ anonymity. In contrast,if scrutinized nodes use a local anonymizing service instead of a third party, then their adversary’s searchspace may be significantly smaller.

The choice between a dispersed and a local anonymizing network corresponds to the motivation forusing anonymity in the first place. There is a big difference between wanting to minimize the chances thatthe adversary suspects communication with a scrutinized node and minimizing the chances that the adversarycan prove the communication took place. (Accordingly, Reiter and Rubin place these notions at oppositeends of their anonymity scale [80].) The dispersed schemes are very good at preventing suspicion, however,once a suspicion is made, much more compelling evidence may follow. For example, if Alice contactshttp://www.amnesty.org through her own Onion Routing Application Proxy, then an adversaryon Alice’s side of the Onion Routing Network would be unlikely to suspect contact with Amnesty, and anadversary on Amnesty’s side would be unlikely to suspect Alice. But if the adversaries knew about eachother, they could surmise the endpoints together using the forwarder tracking attack after sufficient timingsamples (which may take a long time if the network is very active).

Given that a small local anonymizing network supplies the adversary a smaller search space than adispersed scheme, it is significantly easier for the adversary to formulate a suspicion in a local anonymizingnetwork. It is therefore important to ensure that such suspicions are not easily refined into proofs; otherwise,little is gained by the use of a local anonymizer. We assume the difficulty of producing convincing evidenceof contact as a hallmark of local anonymizers, even if a particular local anonymizing system happens to beunusually large. Therefore a local anonymity scheme most fundamentally offers anonymity in the form ofdeniability and not necessarily in the form of apparent innocence.

To make this point clear, suppose that a grand jury calls a computer security expert to testify aboutinformation contained in network logs gathered from a locally anonymous network. In response to thequestion “In your professional opinion, do the network logs show that Alice contacted Bob?” there shouldbe only one possible answer: “The logs are inconclusive,” even though the expert might unwisely go on tovolunteer “But someone in this network definitely contacted Bob.”

5.2 Ramifications of Local Anonymity

In this section we explore the properties of a system that uses local anonymity, that is, anonymization in thesense of deniability within a single administrative domain.

Chapter 5. Local Anonymity 53

5.2.1 Network Capacity

A major advantage of working within a single administrative domain is that the equipment and network aremaximally predictable. This is not to say that one can predict network load in absolute terms, but rather thatthe quantity and quality of information about the network always depends on having unencumbered accessto it.

This is most apparent in contrast to the situation in a dispersed scheme: although its anonymizingnodes cooperate with each other, they may have little knowledge—or be unwilling or unable to disperseknowledge—of the network connecting the interdomain nodes. In particular, the use of randomized routingin a dispersed scheme also causes the network performance characteristics of a stream (throughput andlatency) to be randomized according to the characteristics of the paths connecting the dispersed forwardingnodes.

It comes down to the human factor. By putting anonymizing nodes in separate domains, one surrenderscontrol over performance and administration to people managing and using a remote network, and equallyimportantly, to the aggregate effects of the anonymizing network’s other traffic. It is no coincidence thatthe Onion Routing First Generation Prototype page [76] warns “Notice: do *not* pull zip, mid, or rar filesthrough us. They tend to kill the whole network for everyone” (October, 1998). Indeed, the protocols wedevelop on the local anonymity model in part II can suffer badly under similar circumstances (see x6.2.6).

But inside a single domain one can examine the baseline network loads and determine what resources areavailable for an anonymizing protocol and even create them if necessary. For instance, local-area networks(LAN) often run an order of magnitude faster than their associated wide-area network (WAN) gateways.This is partially due to the issue of local control; a manager can just decide that it is time to upgrade localequipment from 10Mb Ethernet to 100Mb Ethernet, and go out and purchase new equipment. On the otherhand, the Internet Service Provider (ISP) that provides WAN connectivity for that site would be unwise tochange their own equipment without first exploring the impact on their existing traffic and customers. This inturn trickles back down to local network managers. So changing a WAN is fundamentally a more potentiallydisruptive event than changing a LAN. The economics of the situation also favor accelerated improvementsin LANs: since there are many more LAN endpoints than WAN endpoints, economies of scale reduce thecost of LAN equipment faster than WAN equipment.

This common situation—the abundance of LAN capacity relative to the capacity available to long-distance traffic through a WAN—makes it possible to use expensive countermeasures such as uniform sizing(fragmentation and padding), uniform timing (dummy messages), and implicit addressing. These techniquesall adversely affect throughput and latency, but if the local network is fast and wide they may be nonethelesspractical.

If LAN capacity is not abundant relative to WAN capacity so that these countermeasures are infeasible,then local anonymity may be infeasible as well (or at least extremely risky).

5.2.2 Administration

Each dispersed anonymizing scheme superposes a meta-LAN onto a WAN for the sake of anonymity alone.In exchange for making initial suspicions difficult, the administrators have to cooperatively manage a log-ically distinct network and its topology, performance, and failure modes. Such an undertaking is fraughtwith real-world network management concerns.

Perhaps the major issue is troubleshooting. The network distance between anonymizing nodes makes


temporary outages likely, and the fact that “someone else” manages them decreases confidence that theywill remain stable. Following up sporadic outages can be difficult enough when traffic logs are available,but without traffic logs—it seems somehow dishonest to keep traffic logs of an anonymizing service withoutbeing considered an adversary—it is extremely problematic.

In contrast, a local anonymizer puts the entire extent of anonymization under administrative control.Detecting networking problems that affect anonymization is not much different than detecting ordinarylocal networking problems; at least one has access to the equipment in question. If changes need to made toaccommodate the traffic, then it will probably be easier to obtain permission under a single administrationthan if multiple administrations need to cooperate.

5.2.3 Inconspicuousness

Once one accepts the notion that the knowledge of communication partners over time is valuable and wor-thy of protection, it is a small step to concede that the knowledge of whether an anonymizer is in use isalso valuable and worthy of protection. Unless network anonymization becomes commonplace, the merepresence of recognizably anonymous traffic is likely to attract unwanted attention. Another way to say thisis that people become suspicious when they see someone wearing a mask.

Dispersed anonymizers essentially advertise the anonymity of their traffic because their gateways arewell-known. For example, if a web site receives a client connection from www.onion-router.net, itcan probably figure it is dealing with an anonymous connection. Once the forwarding nodes are initiallyidentified, it is a simple matter for a well-placed passive adversary to harvest the addresses of many unrelatedscrutinized nodes. (However, it might not be a simple matter for the adversary to become well-placed.)

In contrast, a local anonymizer has its anonymous/disclosed gateway in the same administrative domainas the scrutinized nodes. Although a well-placed adversary can still harvest scrutinized node addresses,it first has to know to look on that network. Under the assumption that the value of anonymity throughdeniability is realized in long-distance communications, there is little a priori cause for a long-distancecommunication partner to suspect that traffic has been anonymized at the opposite end.

It is important to note that inconspicuousness implies compatibility. It follows that an anonymous/disclosedprotocol gateway in an anonymizing network could can be a scrutinized node of another anonymizing net-work. Crowds and Onion Routing offer protection only to nodes that participate as anonymizing forwardersin the network; scrutinized nodes connecting to the forwarders are completely unprotected. Local anonymitywith its nearby gateway could then be used to protect the links between scrutinized nodes and the dispersedsystem’s forwarding nodes.

5.2.4 Responsibility

One argument for network anonymity goes like this: “Anonymity is the default mode. When you walkinto a store, the clerks do not usually know who you are, and they have no way of finding out. Therefore,we should demand the same from computer networks.” There is only some truth in this analysis. Maskedpeople are not only inherently conspicuous, but they also behave differently than unmasked people, even—orespecially—among complete strangers.

Abuse has been a difficult problem in deployed anonymizing systems. The Penet remailer was ultimatelydeactivated because of abuse; after a copyright was violated by an anonymous post, the authorities tookaction against the remailer owner [61]. This case is confusing because it has many strange facets: the notion

Chapter 5. Local Anonymity 55

that religious scripture can be copyrighted at all, the bewildering aggressiveness with which the copyrightowner pursued its case, the seismic culture clash between lawyers and netizens, and the international scopeand several jurisdictions involved. Still, in the final analysis, one party took offense at anonymous messagesand attacked the anonymity infrastructure in response.

This is not an isolated reaction; other remailers have closed because of abuse. Raph Levien, who keepsstatistics on remailer availability, estimated in 1994 that individual remailers last only about six months onaverage before being shut down by offended users and tired system administrators [63, 51]. Anonymizer,Inc.’s policy of keeping verbose logs is clearly intended to control abuse. At a destination site’s request, theywill even program the Anonymizer to block client requests to anonymously contact that site [1]. This allowssites actively interested in tracking their clientele to easily opt-out of the anonymity mechanism altogether.

As spam, harassment, and unsolicited commercial email become more prevalent, the administrators ofexisting dispersed anonymity schemes spend more and more time defending themselves, their ISP, and theirusers against charges of abuse stemming from their networks. We predict that unless the administratorsfind new ways of controlling abuse, they will find their traffic blocked from networks whose administratorsdo not share their zeal for or even tolerance of anonymous expression. The most straightforward optionavailable to an administrator weary of a particular strain of abuse is to block the abuser’s anonymous serviceentirely—along with all of its non-abusive members.

Some have proposed charging for anonymous service, where the payers can pay with anonymous elec-tronic cash. While this may control the flow of idle, unfinanced spam, we do not see it rescuing the servicesfrom abuse. Wealth and the propensity to abuse seem to run along orthogonal axes [55].

Much has been written about the promise and peril of anonymous messaging [50, 87, 60, 92, 53] insystems where the adversary’s search space is very large. These dispersed anonymity schemes consider alltraffic equally worthy of protection. As anonymity administrators concede that some people do use ano-nymity as an untraceable offensive weapon, they react by weakening the schemes to a level where users areexplicitly threatened to behave responsibly, for instance, by keeping incriminating logs of system use. Theanonymity providers then have no choice but to become arbiters of acceptable and unacceptable demeanor—hardly the role they or their anonymous clients expected them to don.

In a locally anonymous network, the landscape is fundamentally different. Users join in a coalitionbecause they essentially support each other even though they are uninterested in their respective day-to-day details. To the outside world, they appear as one entity. If the unit becomes abusive, there is onlyone set of addresses to block and other anonymous groups are unaffected. Even to accuse the group ofmisbehaving requires addressing all of the members, which gives them an opportunity to revisit the termsof their cooperation and disband if necessary. To insiders, they appear as a tightly-knit group willing toshare responsibility for their network use. The proper analogy is no longer a masked shopper but rather theimpossibility of shopping alone.

We believe that local anonymity encourages responsible network use. It lends a certain amount of formaldeniability to coalition members, not immunity, and users may hesitate before implicating their colleagueswith truly heinous conduct. Those who do not hesitate will soon find themselves without a functionalcoalition.


5.3 Applications of Local Anonymity

Local anonymity can be used in any network with sufficient network capacity to implement appropriatecountermeasures. However, there are certain situations that fit the model particularly well. The commonthread in the following examples is that the adversary is assumed to start from a small search space, i.e., theadversary is already familiar with the scrutinized network. Local anonymity then makes it difficult for theadversary to exploit that knowledge.

� A company could use local anonymity to protect itself from eavesdropping by a subcontractor provid-ing on-site network infrastructure.

� Individuals in a networked apartment building or dormitory could use local anonymity to hide indi-vidual interests and habits from their ISP (and each other).

� Members of a political body such as the U.S. House of Representatives could use local anonymity toprovide shelter from the media, who already use sophisticated technologies to scrutinize their publicand private behavior [95, 64].

� Similarly, a celebrity and her staff could use local anonymity at the office to prevent tabloids andother rumor mongers from distinguishing the celebrity’s shocking interests from her staff’s shockinginterests.

� Dolev and Ostrovsky [32] suggest a scenario where a network connects many possible military com-mand sites, only one of which is ever truly active, and the anonymity is used to prevent the enemyfrom learning the present location of the commanders.

� Laptop users can communicate anonymously without requiring the cooperation of any authorities.(Picture an ad hoc wireless network in a coffeehouse.)

Local anonymity can also be used in situations where a coalition provides a service, i.e., listens for inboundconnections:

� Radically different Web services could be offered at a single site in order to directly protect its clients.As an example, the U.S. Congress could host a web site on a locally anonymous network along withthe secure web site of a foreign political party supported by Congress but unpopular or even illegalabroad. Then remote observers would be unable to distinguish accesses to the Congress site from ac-cesses to the illegal site by examining endpoint addresses alone. Furthermore, this disguises accesseswithout requiring the clients to install special software or hop through general-purpose anonymiz-ing sites. In effect, this form of local anonymity allows the powerful to lend their protection to thevulnerable when they deem it necessary.

5.4 Summary

In this section we introduced local anonymity, a new model of network privacy. Its primary characteristicis that it occurs within a single administrative domain. Secondary characteristics that follow naturally andmake the notion reasonable include network-intensive countermeasures to prevent well-informed adversariesfrom turning observed communications into correct identity conclusions (i.e., an emphasis on deniability),simplified administration, inconspicuousness, and possibly more responsible network behavior by its users.

Part II

Design and Implementation of the NewProtocols

Chapter 6

Introduction to the New Protocols

As part of this research we implemented two local anonymizing protocols. We had three goals that motivatedthis effort. One was to illustrate the local anonymity technique in specific enough terms so that a reader couldencounter the decisions that are faced when contemplating a real system. Building an implementation is anhonest way to make sure these issues arise. Another goal was to determine whether local anonymity and itsnetwork-intensive countermeasures are prohibitively expensive. In our opinion, the implementations plainlyshow that local anonymity is within reach of off-the-shelf computers and networking equipment withoutheroic optimization efforts. Finally, we hoped to experience firsthand the reason that superposed sendinghas been neglected by the implementation community.

Each of the WWW protocols (HTTP, NNTP, FTP, SMTP, IMAP, DNS, RealAudio, etc.) is carried byIP1. Furthermore, every deployed anonymizing technology described in the literature anonymizes a protocolthat is built upon IP (Chapter 4). These observations suggest that IP could be a good candidate for directanonymization. It is low enough in the protocol stack to provide a measure of inheritable protection forcommon protocols including Sun RPC, NFS, telnet, SSH, and IRC in addition to the WWW protocols.Attempting to anonymize even lower protocol layers such as Ethernet suffers from diminishing returns,since IP can be—and routinely is—carried on other types of low-level protocols like SLIP, PPP, FDDI,ATM, and the other IEEE 802.x family protocols. And these low-level protocols (including Ethernet) aremost often implemented in specialized hardware that would be difficult and expensive to modify.

Our two protocols anonymize IP packets in such a way that every IP packet generated by the runningsystem passes through the anonymizing layer. The first protocol is called the trusted gateway protocol (TG),since it relies on a single gateway node (its protocol leader) that reveals sender addresses if infiltrated. Thesecond protocol uses superposed sending (SS); although it therefore has no comparable single point ofvulnerability, it nonetheless designates a protocol leader for synchronization purposes. Both protocols useimplicit addressing for inbound packets and provide client and server anonymity.

We mostly did not attempt to extend anonymity to layers above IP (cf. x2.6, 3.6.3). Therefore, theheaders in an SMTP mail message generated by sendmail still plainly identify the sender’s host name andlogin, NFS transactions still contain the uid of the client, and browsers still volunteer information about theOS in HTTP transactions—all we have done is to make sure this information is not reflected in the headers ofthe IP layer. However, we did make some modifications to the TCP layer to enable meaningful anonymous

1Although some of the WWW protocols are defined generally enough to allow their transport by protocols other than IP, thispossibility has been essentially ignored by implementors.

60 Chapter 6. Introduction to the New Protocols

interactions with external sites. These are described in x8.2.

6.1 Protocol Basics

Our protocols provide client anonymity and server anonymity across the gateway with respect to internaland external adversaries. They also provide rendezvous anonymity when both endpoints are nodes in thescrutinized network.

Each scrutinized node in our protocols responds to the IP address of each other scrutinized node. Inother words, the nodes’ IP addresses are interchangeable outside of the scrutinized network. However, eachnode does maintain a native IP address for Ethernet address resolution [69] and IPsec security association[5] identification. This allows the scrutinized nodes to securely and unambiguously communicate point-to-point in order to execute the anonymizing protocol. Thus our protocols anonymize IP packets, while theanonymizing protocol itself is also carried by IP.

We distinguish between two major types of IP packets. Packets generated by scrutinized nodes as partof the anonymization process are internal. Internal packets are one-to-one or one-to-many communicationsbetween scrutinized nodes. External packets are those formatted for communication with nodes on theother side of the gateway, i.e., the “inconspicuous” packets of x5.2.3. Perhaps unintuitively, internal packetsoften carry external ones in their payloads.

Similarly, we draw the internal-external boundary around those nodes that participate in the anonymousprotocol. The internal nodes are also known as scrutinized nodes.

6.1.1 Outbound Packets: TO GW

In both protocols we use uniform sizing via fragmentation and padding, uniform timing via dummy mes-sages and one-packet-per-round synchronization to provide sender (outbound) anonymity. When a nodewishes to transmit an external IP packet, it encapsulates it and delivers it to the leader node in a protocol-specific anonymity-preserving way. This is called a TO GW (to gateway) operation. When the leader is readyto act upon the TO GW request, it extracts the encapsulated external packet and emits it.

When building the external outbound IP packet, the internal node must fill in its IP source (sender) andIP destination (receiver) address fields. Choosing the IP destination is straightforward: the node simply fillsin a valid IP address for the desired destination node. On the other hand, the node must choose the IP sourceaddress in a way that is independent of the its own native IP address, since the packet being built will betransmitted in the clear when it emerges from the gateway. In our experiments we had each internal nodealways put the protocol leader’s native address in the IP source field. Other choices are possible, but notethat higher-level protocols such as TCP and UDP require coherence in the addresses that comprise a stream[74, 71]. We revisit this topic in x8.1.

6.1.2 Inbound Packets: TO LANON

When a gateway receives an external packet, the gateway makes it internal by adding a TO LANON2 headerand multicasting it to the entire internal network. Each internal node receives the packet applies an implicittest to see if it is being addressed. The test performed depends on the external packet payload, usually a

2“Lanon” was coined by joining “LAN” with “anon(ymous)”.

Chapter 6. Introduction to the New Protocols 61

higher-level protocol. In our experiments the test was straightforward: if the internal node has an opensocket that dictates a response to the packet, then the node responds; otherwise, it silently drops the packet.

6.1.3 Internal Authentication

Every internal packet is nonced (with sequence numbers) and digitally signed, providing authentication,integrity, and anti-replay protection. Under the appropriate cryptographic assumptions, no adversary canundetectably tamper with any internal packet (and therefore the anonymizing protocol) without knowledgeof the cryptographic keys. This dispenses with most active attacks (but see x6.2).

6.1.4 Example

Figure 6.1 illustrates these features with an outbound connection request followed by a response. The arrowsdenote the logical packet flow, but the packets physically flow through the depicted wires and Ethernetswitch.

Node 10.0.0.33 decides to send the first packet in an outbound connection request to 208.8.76.121 (www.oic.gov) and waits for access to the protocol leader node 10.0.0.34. When node 10.0.0.32 gains access,it has nothing to report so it (1) sends a dummy message to the leader. When it is 10.0.0.33’s turn, it (2)sends its external packet bearing source address 10.0.0.34 as a TO GW to the leader. The leader extracts theexternal packet from the TO GW and (3) sends it on to the addressed peer.

Some time may elapse before the peer responds. The three internal nodes exchange dummy messages orprocess other communications during this time. When the peer responds to 10.0.0.34 in step (4), the leader(5) multicasts the inbound external packet as a TO LANON operation. Only node 10.0.0.33 has interest inthe response; the other two internal nodes throw it away.

All of this happens while an adversary is tapped into the switch physically connecting each of theinternal nodes. It can observe or alter packets, but it does not have any of the cryptographic keys in use.The messages (1) and (2) have identical surface characteristics and are equally unintelligible. Messages (3)and (4) predictably show that the protocol leader is the source of the communication. And message (5) ismulticast, so no conclusions can be drawn about the forwarding of (4) in (5).

This example illustrates client anonymity. Server anonymity takes place the same way, but with mes-sages (4) and (5) preceding (1)-(3).

Rendezvous anonymity happens in the same framework; the only difference is that a peer’s address isan internal node’s native address, so it is implicit as well. The internal client prepares an external packetaddressed to its own native address (or any other internal node’s native address, since they are equivalent inexternal packets) and then submits it as a TO GW request. When the leader emits the external packet, it isimmediately multicast to all nodes as a TO LANON packet. If another node responds, then the rendezvouswas successful.

6.2 Shortcomings of the Protocols and Implementation

This is a research implementation constructed in order to focus on performance issues, and it has manyrough edges. As the following sections make clear, this implementation is a testbed only.


10.0.0.34

10.0.0.3210.0.0.33

Peer209.8.76.121

Hm,connection

from10.0.0.34

Adversary

Protocol Leader

It looks like10.0.0.34,

but...

Adversary Tap

Internet

(2) TO_G

W from

10.0.0.34 to

209.8.76.1212 TCP SYN

(1) T

O_G

W d

umm

y m

essa

ge

(3) from 10.0.0.34 to

209.8.76.121

TCP SYN(4) fr

om 209.8.76.121 to 10.0.0.34 TCP

SYN ACK

(5) T

O_LANON

from

209

.8.7

6.12

1 to

10.0

.0.3

4 TCP S

YN ACK (5) TO_LANON

from 209.8.76.121 to

10.0.0.34 TCP SYN ACK

Figure 6.1: A 3-node local anonymizing protocol demonstrating client anonym-ity. Node 10.0.0.33 sends a TCP SYN (connection request) to node209.8.76.121. The adversary is unable to see that the packet came from10.0.0.33 or that the response goes to 10.0.0.33 in spite of its full accessto each packet exchange via the accomplice switch.


6.2.1 Inbound Packet Size Limitation

IP datagrams may be up to 65535 bytes long. Due to an implementation error, our protocols fail to deliver in-bound IP datagrams that are larger than about 65470 bytes. (The precise figure depends on the anonymizingprotocol in use.)

6.2.2 Key Management

We have ignored problems associated with key distribution by assuming that all cryptographic keys aredistributed in advance. This considerably simplifies the implementation but is obviously unworkable forreal networks. However, since our keys are all associated with native IP addresses and never the equivalenceclass of anonymous IP addresses, standard key management techniques apply [82, 57].

6.2.3 Membership Changes

Our implementation reads the internal network topology and appropriate keys from a configuration file atboot time. We did not investigate mechanisms to change the network while it is running, even though this isan important issue in anonymizing protocols.

In general, adding nodes to an anonymizing network improves every node’s anonymity because it en-larges the adversary’s search space, and removing nodes from an anonymizing network similarly degradesanonymity. An important question is what should happen when an internal node suddenly stops participat-ing in the protocol. Since this degrades the security of the remaining network, manual intervention may berequired. This kind of decision involves tradeoffs between the security of the network and communicatingthe security of the network to its users, and has not been studied much as far as we know.

6.2.4 Anonymized Network Layers

Our protocol implementation directly anonymizes the IP layer only; in fact, it really only has coherentaccess to the IP layer (see x7.3.3). In order to provide a kind of TCP and UDP anonymity we have alteredthe internal nodes’ OSes to allow for alternate source IP address selection, but the extension of anonymityto the upper layers is incomplete.

In particular, management of the TCP and UDP port space is totally uncoordinated. For instance, iftwo nodes attempt to build outbound TCP connections with the same source label (IP address and TCP portnumber)—this is allowed by our protocols—then the connections will collide in the TCP layer and bothconnections will be terminated with a TCP reset. Similarly, two internal nodes that listen at the same TCPport address will inevitably destroy each other’s inbound connections should any arrive. And packets thatdo not appear to belong to any active connection in the network do not signal an error but are dropped (seex8.2).

We did not change the OS’s allocation of ephemeral TCP port numbers from the default “use the nextavailable” rule that is applied independently at each internal node. This gives an adversary a very easy wayto sequentially track a node’s outbound TCP or UDP connections.

Although our protocols regulate access to the anonymous gateway in different ways, both protocols cancause queueing in transmitting vulnerable nodes. One might argue that delays consequently incurred shouldbe treated differently than delays due to congestion in ordinary routers: each internal node can tell whenits packet has crossed the gateway, but nodes normally cannot track a packet’s progress through external


routers. Our implementation has no feedback mechanism for this information. Therefore, TCP timers areset according to the time the packet leaves the TCP layer and not the time the packet crosses the gateway.

6.2.5 Reliable Multicast Assumption

Our protocols use multicast to distribute inbound messages and initiate communication rounds. While in-tegrity protection prevents an active attacker from altering these messages, we have no mechanism to guar-antee message delivery. Our use of sequence numbers allows the network to subsequently detect that internalpackets have disappeared, but by then the adversary’s damage may already be complete.

Suppose an adversary controlling the network switch suspects a node of being the endpoint of a TCPconversation in either an SS or TG network. The adversary can fabricate a TCP FIN (finish connection)segment and send it to the leader, capture the resulting TO LANON that encapsulates the FIN, and deliverthe TO LANON to every node on the network except the suspected one. The receipt of a FIN on an activeconnection causes a FIN ACK to be generated. So if the suspected node is not the endpoint, then the network(i.e., another node) will quickly generate a FIN ACK, otherwise no FIN ACK will appear on that connection.This attack is applicable to TG networks as well as SS networks.

Waidner [96] describes how to modify the SS protocol to immediately (i.e., in the next round) detectmulticast failures. His suggestion is to modulate every node’s symmetric keys in each round by a functionof the recently received multicast data. If every node receives the same multicast data, then the modulationswill all be uniform, so everything will keep working. If one node receives different multicast data than theothers, then its keys will not agree with other nodes’ keys, so it will cause immediate inconsistencies andfailures; in any case, the anonymizing network will stop carrying data. Waidner therefore calls this “fail-stopbroadcast”.

This idea is somewhat complicated in our environment, because our TO LANON operations arrive andare redistributed by the leader asynchronously. This means that in general, the internal nodes may correctlydisagree as to which keys are most current. We can compensate by introducing some looseness into thenotion of “current key”.

We have not implemented these techniques. Therefore, our protocols are only appropriate for use onnetworks that reliably disperse multicast messages.

6.2.6 Multicast Limits Scalability

Since our protocols use multicast both to initiate a communication round and to distribute inbound messages,their performance is limited by the network fabric’s multicast scalability. In addition, every inbound packet isintended to be received by every internal node, but almost all of the nodes will ultimately ignore almost everyinbound packet. So first, as the network size grows, our protocols strain the network fabric. And second,assuming that the amount of inbound traffic grows with the network size, the CPU load on participant nodesgrows as well.

Our experiments with up to 16 nodes did not seriously stress the network switches we used (see thenetwork utilization sections x9.5.4 and x10.9.5), so we did not experience the first limit. And since wedid not run experiments that included significant simultaneous inbound transmissions (see x7.4.1), neitherdid we experience the second limit. Nonetheless, we feel that this second limit is in fact significant in ourimplementation, and that it requires further investigation.


6.2.7 Fault Tolerance

Our protocols do not specify what should happen in response to failure. Our implementation generally viewsfailed assertions as implementation errors and halts the network for debugging. Pfitzmann [67] describesa sensible mechanism for automatic error localization in Superposed Sending networks. This is describedmore fully by Niedermaier [62].

6.2.8 Denial of Service Attacks

The local anonymity technique does not recommend any denial of service countermeasures, nor have weimplemented any. We assume that participant nodes are friendly and that failed assertions and other unex-pected events are implementation errors rather than possible attacks. Therefore, these events usually causethe network to halt for debugging purposes.

While we are not aware of degenerate behavior such as crashes or memory leaks resulting from nodemisbehavior or active attacks, we did not separately review the implementation for such vulnerabilities.

The use of superposed sending carries a special risk. Since each node’s participation is required toproduce a single SS round’s output, any participating node can untraceably disrupt the protocol at will. Theliterature [18, 14, 98] describes techniques to minimize this risk, but we have not implemented any of them.Our SS networks remain vulnerable to disruption attacks by infiltrated internal nodes.

6.2.9 Attacks Based on Other Characteristics

At least one measurable characteristic of an internal node is apparent in spite of the anonymizing protocol.We have found that a node’s response time to an internal anonymizing packet increases with increasing CPUload (see x10.9.4). This might allow a patient and well-informed adversary to infer which nodes are merelyrunning the anonymizing protocol and which nodes have competing computation (i.e., are not idle). Butthis is an implementation artifact that can be alleviated by coding the protocol in a sufficiently high-priorityscheduling domain.

Chapter 7

Experimental Platform Description

7.1 Computing and Networking Equipment

In protocol development and the experiments we used up to 17 Pentium-class PCs and up to three 12-port 3Com SuperStack II Switch 3000 10/100 Ethernet switches. The equipment was connected using thephysical topology illustrated in Figure 7.1. For an experiment of n internal nodes, we used the nodes labeled0::n from Table 7.1 with node 0 running strictly as an experiment controller and disclosed-protocol peer.

Referring to Figure 7.1, this means that all experiments of 6 or fewer internal nodes took place on asingle switch, so point-to-point communications were essentially collision-free. Once the 7th node wasadded, its communication with the other nodes had slightly higher latency due to the extra hop betweenswitches. Once the 8th node was added, nodes 7 and 8 had to share the bottleneck link between switches 21and 20, and so on.

The internal nodes were also equipped with 128 MB RAM and ample disk space, but we observed noexcessive memory consumption by the anonymizing network stacks: the protocol leader (node 1) consumedbetween 2-3 MB depending on the protocol in use, while the other internal nodes consumed about 2 MB.These figures were stable and independent of the network load or experiment in progress. We expect thatour experiments would produce identical results on PCs equipped with 16 MB RAM.

7.2 Operating System

The computers used in the experiments ran a modified Linux kernel version 2.0.30. The rest of the systemsoftware (applications, daemons, libraries) ran Red Hat [79] version 4.0 and version 5.0. The protocols andkernels were developed on a system running Red Hat version 4.0.

These choices were seen as (perhaps minimally) consistent with our goal of evaluating system perfor-mance with typical off-the-shelf components. While Linux only accounts for a small fraction of installedcomputers, it is a widely understood open-source operating system and seen as a mainstream implementa-tion of Unix. We did not possess sufficient knowledge, experience, or source code to investigate a Windows95 or Windows NT implementation.

68 Chapter 7. Experimental Platform Description

1 2 3 4 5 6

7 8 9 10 11 12

13 14 15 16

0

Switch 21

Switch 20

Switch 22

Internet

Figure 7.1: Physical network topology in the experiments. System 0 was the ex-periment controller and had the only direct connection to the internet.The switches were “stacked,” creating a virtual LAN.

Chapter 7. Experimental Platform Description 69

Number Name Manufacturer CPU MHz Network Interface

0a St-Marys Dell Pentium II 266 DEC Tulip1b Riverway Dell Pentium Pro 200 DEC Tulip2 Storrow Dell Pentium Pro 200 DEC Tulip3 Berkeley Dell Pentium Pro 200 DEC Tulip4 Backbay Dell Pentium Pro 200 DEC Tulip5 St-Paul HP Pentium Pro 200 3Com 3c905b6 Buick Dell Pentium II 233 3Com 3c905b7 Masspike Dell Pentium II 233 3Com 3c905b8 Brookline Dell Pentium II 233 3Com 3c905b9 Cummington Dell Pentium II 233 3Com 3c905b

10 Granby Dell Pentium II 400 3Com 3c905b11 Queensberry Dell Pentium II 400 3Com 3c905b12 Baystate Dell Pentium II 233 3Com 3c905b13 Bowdoin Dell Pentium II 400 3Com 3c905b14 Tremont Dell Pentium II 400 3Com 3c905b15 Huntington Dell Pentium II 400 3Com 3c905b16 Boylston Dell Pentium II 400 3Com 3c905b

aExperiment ControllerbProtocol Leader

Table 7.1: Computers used in the experiments. The Ethernet cards ran at 100Mb/sec in half-duplex mode. (The systems are named after streets nearBoston University except for Backbay, which is a neighborhood.)


7.3 The x-kernel Networking Platform

The anonymizing protocols are implemented in version 3.2 of the x-kernel [43], an efficient and portableplatform for network programming.

We chose this platform instead of the native Linux kernel for a number of reasons. First, the x-kernelprovides efficient and well-documented implementations of a number of standard protocols including TCP,UDP, IP, Arp, and Ethernet. An implementation of some of the IP security protocols (IPsec) was alsoavailable [45] (with its cryptographic components subject to the usual U.S. export restrictions).

Second, protocols in the x-kernel adhere to a Uniform Protocol Interface and consistent conventionsas much as possible; this allows a programmer to assemble the protocols into a network stack at compiletime. This has practical as well as research and pedagogical value. For example, we often compiled theanonymizing layer into a network simulation stack. Instead of the Ethernet protocol layer being connectedto an Ethernet-capable network interface controller (NIC), it was connected to the test system’s native UDPlayer, allowing one system to simulate multiple nodes by multiplexing within the UDP port space. Thisresulted in simulated mini-experiments in which protocol behavior was much easier to control and observethan in real multi-node experiments.

Finally, we anticipated it would be very important to be able to run the protocols within a rich debuggingenvironment. Indeed, our protocols underwent substantial revision in response to problems uncovered bydebugging. The x-kernel was helpful in this regard because it can be realized as an ordinary, debuggableuser-space process in the Linux environment1. In contrast, networking stacks are often implemented at leastpartially in the OS kernel for efficiency and because the OS relies upon on networking capability.

7.3.1 Threading in the x-kernel

The x-kernel is multithreaded with a message-per-thread model, that is, a single thread carries a messagebetween the highest-level and lowest-level protocols in a network stack. In our environment, the threading isaccomplished completely in user-space by a Pthreads-compatible [70, 16, 52] implementation [26] originallydue to Provenzano [77]. It runs the threads within a single Unix process, using interval timers, signalhandlers, and direct (nonportable) user stack manipulation for context switching. Unix system calls that arecapable of blocking the calling process are meticulously proxied by the threading package and translatedinto nonblocking equivalents.

7.3.2 The Anonymizing IP Protocol Graph

Figures 7.2 and 7.3 shows the graph of the x-kernel protocols that we used to anonymize IP packets. Theanonymizing protocols were mostly confined to the layer labeled ip/0, although minor modifications tosome other protocols were necessary.

7.3.3 The Ethtap Interface

We used the ethtap kernel module [45] in order to attach the x-kernel to a live Linux system. Ethtap isa standard component of the Linux version of the x-kernel that provides hooks for observing and modifying

1The x-kernel was originally constructed as an operating system networking framework with both kernel-space and user-spacecomponents. However, the implementation we used was realized entirely in user space (except for ethtap, see x7.3.3).


ethkern

eth/1

arp/1

vnet/1

ip/1

ipf_udp ipf_tcp ipf_icmp

Linux NetworkStack

ip/0

Figure 7.2: The protocols in this half of the graph are primarily glue connectingthe anonymizing layer ip/0 with an active Linux kernel. The rectan-gles are user-space x-kernel protocols, and the cloud is in native Linuxkernel space. Figure 7.3 shows ip/0’s “downward” connections.


rfc1825

tcryptkm/a md5 crypt3in idea

ofbprsg

ip/0

km/dc

sink km/a km/b km/alg km/md5 km/crypt km/crypt3 km/dc km/idea

vnet/0

km/tcrypt

arp/0

eth/0

ethdev

saidmgrkm/crypt3in

Ethernet NIC

crypt crypt3

Figure 7.3: Most of the complexity in this half of the protocol graph is due to IPsec(rfc1825 [5]) support. The rectangles are user-space x-kernel proto-cols, and the cloud is in native Linux kernel space.


a system’s native Ethernet traffic. These hooks appear in the “connection bolts” at the bottom of Figures 7.2and 7.3. The next two sections will clarify ethtap’s role in the system.

7.3.4 Outbound Packet Processing

The running system contains a complete Linux network stack, so applications and the Linux kernel naturallygenerate network requests that ultimately appear as Ethernet frames. Right before these Ethernet frames arehanded over to an NIC for transmission, they are intercepted by ethtap’s “high tap” and diverted into akernel buffer pool. Running threads of the user-space x-kernel protocol ethkern in Figure 7.2 meanwhilepoll this buffer pool via system calls. When a thread finds a frame, it shepherds it upwards through theprotocol graph. Non-IP frames are discarded at the vnet/1 layer, and IP packets that are not UDP, TCP,or ICMP are discarded2 at the ip/1 layer. The appropriate ipf * layer then bounces the packet backdownwards towards ip/0.

At the ip/0 layer (see Figure 7.3), the packet is anonymized according to the current protocol state.This may involve pushing the packet down into one of ip/0’s four right-hand “leaf” protocols; this wouldcause the packet’s data to be transformed and immediately returned to ip/0. Once ip/0 builds a packet toput on the wire, it pushes it through vnet/0; the packet flows to eth/0 (where it gains Ethernet headers),and then to ethdev. Ethdev uses a system call to send the packet through ethtap’s “low tap”, where itis then delivered to the NIC as a transmission request (hard start xmit()). The thread’s calling stackthen unwinds, possibly all the way to ethkern where outbound packet processing began.

7.3.5 Inbound Packet Processing

Inbound packet processing runs in the opposite direction. When the system’s NIC receives an Ethernet framefrom the wire, the frame is intercepted by the “low tap” of the ethtap kernel module and sent to a kernelbuffer pool. Running threads of the ethdev protocol in Figure 7.3 poll the buffer pool via system calls.When a thread finds a frame, it shepherds it upward through the protocol graph, where it may ultimately bebounced off of an ipf * protocol in Figure 7.2 and back down towards ethkern. This time, ethkernwrites the packet via a system call to ethtap’s “high tap”. This causes the packet to be registered in theLinux network stack as “just arrived from a NIC” (netif rx()).

Note that TCP, UDP, and ICMP are missing.

7.3.6 Discussion

With a thread scheduling granularity of 10ms, there is enough time for a NIC to receive about 80 maximum-size Ethernet packets between timer events on a 100 Mb/sec Ethernet. Ethtap takes advantage of this bydelivering bursts of packets from its buffer queues when one of its taps is polled, thus reducing the numberof domain crossings.

Ethkern and ethdev do not burst transmissions to ethtap; each packet must be transmitted indi-vidually. Although transmitting requires a boundary crossing, the Linux kernel does not run its scheduleruntil the system call returns; by then, the NIC has already begun transmitting the packet on the wire.

2This means that only UDP, TCP, and ICMP packets are anonymized or even processed in our system. This is merely anidiosyncrasy of our implementation and has nothing to do with the local anonymity technique.


Per-packet boundary crossings are neither ideal nor typical for a Unix system. In our experimentalenvironment, a single inbound packet requires several boundary crossings. First, a frame arrives and theNIC generates an interrupt. In the interrupt “bottom half” (a natural scheduling boundary), the kernel putsthe frame on the low tap read queue and returns. When the x-kernel is scheduled, a user-space threadrunning on behalf of the ethdev protocol polls the kernel buffer via a system call (1) and shepherds3

the frame through the x-kernel protocol graph. If the protocols decide to deliver the packet to the runningsystem, they reformat it as an Ethernet frame (in eth/1) and push it through ethkern to write it to thekernel via a system call (2). The situation is similar when the Linux kernel attempts to transmit a frame.

In both cases, the packet suffers two more boundary crossings than it would in an ordinary Linux sys-tem. Furthermore, both types of kernel buffer are accessed by polling threads, meaning that ethdev andethkern cause the x-kernel to absorb all available CPU time. Since the x-kernel uses nonpreemptablethreads, the polling threads do not burden the protocols once they are processing a real packet. However,the system saturation by the x-kernel makes it challenging to obtain meaningful CPU usage statistics.

For all of these reasons we consider the numbers reported in Chapters 9 and 10 to be at the low end ofattainable performance. We expect an implementation that is well-integrated into a tuned network stack tobe measurably more efficient.

7.4 Experiment Methodology

The results reported in Chapters 9 and 10 refer to experiments that are executed as follows.

1. The internal nodes (numbered 1 : : : n) listen for instructions from the controller (node 0) using stan-dard system networking. That is, the anonymizing protocol is not yet running on any system.

2. For each internal node i 2 f1 : : : ng:

(a) The controller sends an x-kernel and configuration options to node i.

(b) Node i starts the anonymizing x-kernel. This diverts all of node i’s IP packets into its anonym-izing layer.

3. Now all nodes are running the anonymizing protocol. The controller multicasts the start signal to thenetwork. At this point the anonymous network has booted.

4. The “body” of the experiment takes place.

5. When the experiment has finished, one of the involved nodes multicasts a “poison” packet. Thiscauses each of the running x-kernels to exit, so nodes 1 : : : n once again become non-anonymizing.

6. Each node 1 : : : n returns a transcript of messages produced during the experiment to node 0.

3We have glossed over an inexpensive thread context switch here, which is apparently a remnant of the original x-kerneluser/kernel space separation. Unexpected thread activations are possible at this point.


7.4.1 The Experiments

We ran four types of experiments on each protocol in order to extract performance data. (We ran one extraexperiment on the SS protocol; it will be explained in x10.) Although the name of each experiment suggestsa particular purpose, we gathered and report performance results from each of them in various ways.

Our experiments are based upon simple maximum-speed unidirectional TCP bulk data transfers. Moresophisticated models such as SPECWeb96 [20] and SURGE [6] exercise a server’s TCP connection manage-ment, memory management, task management, and Web server application software. Our protocols onlyimpact those areas in terms of increased CPU competition, so our experiments just measure how quicklyan anonymizing network can carry data through an unencumbered TCP layer. We feel this is a reasonablemodel of anonymized WWW access dominated by HTTP.

Scalability. The outbound scalability experiments examine how the network’s behavior changes as nodesare added to the network. In the body of the experiment, one node built an outbound TCP connection to anexternal peer and transmitted 204800 TCP bytes as fast as possible, then the connection was closed. Thereceiver side of the connection simply absorbed and discarded the data; the only packets flowing towards theanonymous network were acknowledgment and connection setup and teardown packets. This was repeatedfor a total of 10 iterations.

The application program controlling the TCP connections on both ends was derived from ttcp. Thesender actually transmitted over each connection with 200 writes of 1024 bytes each, but since TCP isnot a record-oriented protocol, this segmentation of the data vanished by the time the external packets werechunked. It is not externally visible.

Contention. The contention experiments measure how the network’s behavior changes when multiplenodes simultaneously attempt to transmit outbound packets. We ran the experiment for network sizes n 2

f1 : : : 12g and number of simultaneous transmitters m 2 f1:: : : : ng. The experiments took place like theoutbound scalability ones except that in experiment (n;m), nodes f1 : : : mg each built an independentoutbound TCP connection, and we only ran 5 sequential iterations of each experiment.

Indistinguishability (outbound). The indistinguishability experiments sequentially rotate a network end-point among the 7 participant nodes. The outbound experiments are essentially the same as the scalabilityexperiments in the case n = 7 except that transmitter changes after every 10 outbound connections.

We chose the word “indistinguishability” because we used these experiments to informally verify thatthe internal connection endpoints were not apparent to internal or external observers.

Indistinguishability (inbound). In the inbound experiments, the external peer created the inbound con-nections while only one of the 7 internal nodes was prepared to respond. After every 10 inbound connections,the internal listener changes.

7.4.2 Trace Gathering

Our performance results are computed from logs of libpcap-format packet traces gathered during theexperiments. Since most internal communications involve the protocol leader, we took the traces on thatsystem alone.


Specifically, we modified the eth/0 layer in the protocol graph of Figure 7.3 to record every inboundand outbound packet it encounters. This allowed us to capture even non-IP packets. Since inbound packetsdo not enter that layer until the x-kernel process and an appropriate thread within the x-kernel are scheduled,this is a source of some inaccuracy in the logs. The Linux kernel scheduling granularity is 10 milliseconds,so an inbound packet may easily be delayed for several milliseconds before it receives a timestamp.

At one point in development we instrumented a separate system to improve the timing accuracy of thetraces, but this approach requires either extremely circuitous routing or the use of a network hub instead ofa switch. Since the x-kernel trace generating technique preserves the order of the packets and is accurate toroughly the scheduling granularity of the system, we were comfortable using it to produce throughput anddelay figures.

In the “tree” experiments (see x10.9.1), internal communications also take place between non-leadernodes. We did not gather these packet traces, and so restrict our attention to statistics that describe thenetwork as a whole.

7.4.3 Post-processing and Analysis

To analyze the traces, we modified tcpdump to understand our protocols and wrote Perl programs to parseits output.

7.4.3.1 Throughput

The primary statistic computed from the traces is the anonymous gateway’s TCP throughput, i.e., thetotal number of TCP bytes crossing the gateway per second. Our computation of this figure relies ontcpdump2xplot.pl, a part of the xplot package by Shepard [86]. In our experiments, each TCPconnection is considered one-way: the opposite direction consists solely of TCP acknowledgments and isnot reported in statistics.

We chose to focus on TCP throughput because it is a familiar statistic and probably the most relevantone from a usability point of view. The achieved IP throughput is perhaps a more direct descriptor of ouranonymizing protocols’ performance (they anonymize IP, after all), but results describing how quickly wecan blast raw IP packets through the gateway do not immediately suggest what a typical client is likely toexperience. In other words, we see a typical client as one that is TCP-focused. In any case, TCP throughputis directly related to IP throughput; our concentration on TCP is purely for the sake of the absolute results.Relative results between network sizes and major protocol types are as well described by TCP throughputas they are by IP throughput.

To compute the gateway TCP throughput, we first identify all of the TCP connections present in a trace.We then independently compute each connection’s instantaneous throughput over time, producing exactlyas many throughput samples as there are packets in the trace. We compute the arithmetic mean of thesesamples to produce the connection’s average throughput and sum each connection’s average throughput toreport the network’s aggregate anonymous throughput.

Since we measure only the number of TCP bytes emitted by the gateway over time, TCP retransmits donot reduce the reported throughput. In other words, a TCP retransmit of 1340 bytes is simply counted as1340 bytes emitted during that round. Although the TCP retransmits may be due to delays inherent in ourgateway arbitration techniques, they are strictly a function of the TCP layer and we have not adjusted forthem in any way. Therefore, the average TCP throughput as computed by end-to-end application measure-


ments may differ from the throughput we compute. In practice, the connection peer was always located veryclose to the anonymous network; this minimized the likelihood of retransmission timeouts. We found thatthe two measurements they were not startlingly divergent, but we did not record end-to-end statistics for adetailed comparison.

Another consequence of our measurement convention is that we do not account for the TCP and IPheaders that are also emitted by the gateway. This systematically underestimates the absolute throughput ofour protocols, since the external TCP and IP headers—usually 40 bytes together—have to be anonymizedas well. Accounting for these headers shows that our absolute TCP throughput is actually about 3% better4

than we report.

7.4.3.2 Delay

Since the leader gateways at most one packet per round, each internal node attempting to transmit suffers anaverage delay of half the time required to process a round, assuming there is no contention for the gateway.This figure is easily computed from the packet traces. A more meaningful statistic is the number of roundsa packet is delayed due to contention. We report this figure as well in the experiments that investigatecontention behavior.

7.4.3.3 CPU Usage

As mentioned in x7.3.6, the polling nature of the ethtap architecture makes CPU statistics hard to obtain.But we did gather some data.

Both the packets that the Linux OS transmits and the packets that the NIC receives enter the x-kernelprotocol graph at the bottom via the xDemux function. Tracking the CPU time spent within xDemux istherefore a good approximation of the CPU anonymization effort. It takes most facets of anonymization intoaccount, including parsing packets from the OS or the NIC, encrypting and decrypting, defragmenting, andso on.

To track this time, we wrapped a timer around the xDemux function. After every 100 xDemux calls,we record the average time per call together with the percentage of elapsed CPU time this represents. Sincethe x-kernel coding conventions prohibit thread blocking within an xDemux call, each event so timed onlycounts real x-kernel processing.

But it is possible for the x-kernel process to be preempted by the Linux scheduler before an xDemuxcall completes. This skews our statistics, since we use a real-time clock that keeps running during a processswitch rather than the process time clock to track xDemux. (The process time clock was too low-resolutionto measure the time spent in xDemux.) Our CPU usage reports therefore overestimate the amount of timespent in xDemux. However, manual examination of the data—and the fact that competing processes weregenerally network I/O bound, executing for much less than the allotted timeslice—lead us to view thisinaccuracy as relatively minor.

Activities that do not take place within an xDemux include timer-generated events such as retransmis-sion requests and ethtap polling. But the former do not occur in our experiments after the anonymousnetwork has booted properly, so they are of little consequence. And since polling is due to the use of

4I.e., real throughput = 1:03(reported throughput).


ethtap, it is not unreasonable to discount that cost when describing the protocols—a production imple-mentation would certainly use a different architecture. But do note that successful polls result in packetcopies, and this is not accounted for in our figures.

In summary, we did produce some (somewhat suspect) CPU usage figures. We found that the leaderspent more than 70% of its time in xDemux, while other internal nodes spent more like 10%–30% of theirtime in xDemux (depending on the experiment). This supports the reasonable notion that it is expensive tobe a protocol leader and that the whole network’s performance is CPU-bound.

Chapter 8

Features Common to Both Protocols

Being realizations of the local anonymity technique, our two protocols have much in common and they sharemuch code. In fact, we produced just one x-kernel image corresponding to the protocol graph in Figures7.2–7.3; the protocol to use is determined from a configuration file at network boot time.

8.1 Address Selection in External Packets

The use of implicit addressing allows each internal node to use the name of any internal node to identifyitself in external packets. In our implementation, this choice is actually made in the Linux OS kernel. Thex-kernel expects to be given an external IP packet with its headers already in external form.

The reason for this is that our architecture does not have direct access to TCP, UDP, or other higher-layer protocols. These protocols rely on the IP address choice to identify connections, so the local IPaddress chosen for the first packet of a connection must be used for every subsequent packet. While it wouldbe possible for our implementation to look inside preformatted packet payloads (such as the TCP layer)in order to track connections and adjust the IP headers accordingly, our current strategy completely avoidshaving to examine packet contents. Furthermore, changing IP headers of formatted TCP or UDP packetswould require the recomputation of TCP and IP header checksums.

Therefore we modified the Linux kernel to allow for this choice in the bind sockets call, which takesplace at the time when a TCP or UDP or other socket is being created. The mechanism is derived from oneused to implement server selection and triangle routing in the Distributed Packet Rewriting (DPR) project atBoston University [10]. To specify the space of possible local addresses, we use the rr (randomized route)command:

# rr riverway riverway berkeley storrow

This example says that 2=4 of streams’ local addresses should be the primary native IP address of riverway,1=4 should come from berkeley, and 1=4 should come from storrow. When a bind call takes place, thekernel chooses randomly from the list according to the specified simple biased distribution.

As implied by x6.1.1 and Table 7.1, our experiments always ran while

# rr riverway

80 Chapter 8. Features Common to Both Protocols

was in effect. This simplified trace analysis without changing anonymity properties or artificially inflatingperformance figures. In fact, since riverway was both the protocol leader and the slowest system we used,our performance figures could probably be improved by taking better advantage of the rr mechanism.

8.2 Modifications to the Upper Layers

Our use of implicit addressing for inbound packets causes every node to receive every inbound TCP packet.Of course, most nodes will not recognize most of the inbound TCP packets. When this occurs, an ordinaryTCP/IP stack will send a TCP reset (RST) in order to inform the sender that packets have been misdirected.In our environment, this would cause each inbound TCP packet to elicit a TCP RST from each internal nodethat does not use the connection, quickly closing all open TCP connections.

Our solution was simply to suppress these TCP RSTs. A node receiving a TCP packet that can not beassociated with any active socket silently drops the packet. Other TCP RSTs are unaffected, such as thosesent when sequence numbers are inconsistent or data remains unprocessed during a close.

Similarly, we modified the UDP layer to inhibit ICMP Destination Unreachable messages.This is a simple solution that works well in most circumstances. However, there is an obvious side effect.

An inbound connection attempt on a port on which no node listens is not refused as one normally expects;instead, it is silently ignored. To the disclosed peer, this is indistinguishable from a network failure.

This effect can be eliminated in TCP connections by having one or more of the nodes keep track of allthe connections that are open in the network and sending RSTs when appropriate. But our protocols donot explicitly deliver a copy of each outbound packet to each interior node, and inferring TCP state basedon only the inbound packets would be approximate at best. So eliminating the effect completely wouldeither further burden the gateway, require internal distribution of outbound packets, or not work very well.Besides, there appears to be no anonymity-preserving way to infer that UDP packets are undeliverable: asuccessful UDP receipt need have no external consequences at all.

It should be noted that during each of our experiments we reduced the Linux OS’s perception of theNIC’s maximum transfer unit (MTU) from 1500 to 1380 bytes (a reduction of 8%). This figure was chosenso that TCP packets would not need to be fragmented when all of the headers associated with anonymizingwere appended, no matter what protocol was being used. The reduction was generous; our protocols didnot require more than about 80 bytes of headers. (The x-kernel is aware of the larger MTU, so it can buildEthernet packets up to 1500 bytes long.)

8.3 Address Resolution Protocol

When a node with IP address A tries to transmit to node with IP address B, it must also find a networklayer address to use for the first hop in the internet route. If these addresses are not known in advance,then Ethernet-based systems use the Address Resolution Protocol (ARP) [69] to determine an appropriateEthernet address1. Since ARP is not carried in IP packets, our system does not anonymize it. An adversarycould simply watch for the ARP requests to determine if a node is transmitting, and possibly even discoverthe first hop on the path.

1IP carried on other network types also requires address resolution, and the anonymizing issues are similar.

Chapter 8. Features Common to Both Protocols 81

vershdrlen

type ofservice

total length

identification fragment offset

TTL protocol header checksum

source address

destination address

reserved

don't frag

more frag

payload

0 8 16 24 31

8 16 24 310

Figure 8.1: Format of Internet Protocol (IP) packets. The scale is in bits.

We eliminate this possibility by having each non-leader interior node specify the protocol leader as thefirst hop for every route. The leader uses ARP in the ordinary fashion to determine how to send its externalpackets, but by that time an adversary cannot tell whether the leader originated the packet or one of theinternal nodes did.

There is also a somewhat artificial ARP issue caused by our use of ethtap to attach to a runningsystem. When the Linux OS kernel “transmits” a packet onto ethtap’s high tap queue, it must include anethernet address even though we discard this address as the packet traverses the protocol graph. Similarly,when the x-kernel “transmits” an IP packet to the node’s Linux OS kernel, it must gain an ethernet headereven though the Linux OS kernel does not examine the header. We have rigged routing tables and ARPconfigurations with dummy addresses so that these requirements never cause an ARP request to be generatedand transmitted.

8.4 Datagrams, Packets, Fragments, and Cells

Much of the complexity of this implementation has to do with fragmentation and reassembly. We use thefollowing definitions for clarity: an IP datagram is an IP message [73] (see Figure 8.1) that is unfragmented,i.e., its “fragment offset” field is 0 and it does not specify the “more fragments” flag. An IP packet is anyIP message: either a full datagram or only a fragment of a datagram. A packet that is a fragment of a largerdatagram is naturally enough called an IP fragment.

IP allows fragmentation of its datagrams because the MTUs of the networks along a route vary. A router


attempting to forward a packet onto a network whose MTU is too small can simply split the packet intoMTU-sized fragments and transmit them independently. It is then up to the target node to reassemble thefragments into the original datagram. As explained in RFC 791 [73],

To assemble the fragments of an internet datagram, an internet protocol module (for exampleat a destination host) combines internet datagrams that all have the same value for the fourfields: identification, source, destination, and protocol. The combination is done by placing thedata portion of each fragment in the relative position indicated by the fragment offset in thatfragment’s internet header. The first fragment will have the fragment offset zero, and the lastfragment will have the more-fragments flag reset to zero.

Since we using uniform sizing via fragmentation and padding, every transmission from an internal nodeto the leader has the same surface characteristics. These messages are called cells. A packet (datagram orfragment) may be split across multiple cells. The cell size in use is defined by a configuration file at networkboot time. We used cell sizes close to 1500 bytes in our experiments. Only messages from internal nodes tothe leader are cells; other internal communications have variable length.

8.5 Gateway Contention

The TG protocol allows the leader to accept at most one external IP packet per round, while the SS protocolaccepts up to two external packets in a round. Once the cells of an outbound packet have been collected, theleader emits the packet, possibly fragmenting it again for the outbound network.

Another approach would be to allow the leader to emit up to one external IP packet per round perinternal node. However, that approach would sometimes allow an adversary to determine that multipleinternal nodes were communicating. In the interest of deniability, we decided against the possibility. Inaddition, the mathematics of the SS protocol that we implemented naturally restrict the leader to recoveringat most two packets in any given round.

8.6 Types of Operations

Each internal communication has a field that specifies the operation type. The precise formats are describedin the protocol-specific chapters x9 and x10. The operations in the two protocols are similar, although theydiffer in details.

8.6.1 TRIGGER

The TRIGGER multicast operation originates at the leader and signals the beginning of a communicationround. It also indicates which cells (at most two) were accepted in the previous round. This allows theinterior nodes to free resources associated with their transmission attempts.

8.6.2 POLL

Both protocols also support a unicast variant of TRIGGER called POLL. The POLL operation is used whena node has not responded to a TRIGGER in a reasonable amount of time.

Chapter 8. Features Common to Both Protocols 83

8.6.3 TO GW

(See also x6.1.1.) TO GW cells are used to carry external packets from interior nodes to the leader. Each cellsent also carries a cellID, a randomly chosen 8-bit number intended to uniquely identify the cell within around. These are the identifiers used by TRIGGER operation to indicate which cells won access to the leaderin the last round. Note that the leader only reassembles cells into packets before emitting them; it does notreassemble outbound packets into datagrams.

The protocol leader is also a scrutinized node. It competes fairly with other internal nodes when at-tempting to transmit its own packets, even though it does not actually put TO GW operations on the wire.That is, it is no more likely to have its cells accepted than other internal nodes, and it uses the same cell sizeas the other nodes in spite of the fact that its packets need not actually traverse an MTU-bound network inorder to reach the leader.

It is possible for multiple nodes to randomly choose the same cellID. The two protocols have differentmechanisms for resolving this kind of conflict.

TO GW cells are also used to transmit dummy messages. Dummy messages are referred to as blather inour implementation. The protocol leader will never accept a dummy message if a non-dummy message wassubmitted in that round.

8.6.4 TO LANON

(See also x6.1.2.) TO LANON multicast messages are used to deliver inbound messages to the interiornodes. Any internal node may multicast a TO LANON message to the network. These operations are notsynchronized to the communication round cycle and may happen at any time. In our experiments, theprotocol leader sent most of the TO LANON messages because the scrutinized nodes all used the leader’snative IP address in their external packets (see x6.1.1).

The initial recipient of the external datagram gathers all of the datagram’s IP fragments before forward-ing it in TO LANON operations. This both minimizes the number of TO LANON operations and preventsother interior nodes from possibly having to defragment an extra time to recover the external datagram2.

But a maximum-size IP datagram has no room for extra headers no matter how the datagram may befragmented on the wire. An internal node that receives such a very large inbound datagram therefore cannotrely on the IP fragmentation mechanism deliver the TO LANON operation. Our protocols fail to accountfor this, which is why our implementation does not correctly distribute extremely large inbound datagrams(those longer than about 65470 bytes). This can be fixed by fragmenting first, allowing room for TO LANONheaders each time, and transmitting multiple TO LANON operations rather than fragmenting a single verylarge TO LANON operation.

Note that the x-kernel anonymizing IP layer does not know whether an incoming TO LANON operationis of interest to the node. It simply sends the associated datagram upwards through the protocol graph ofFigure 7.3. The Linux OS kernel eventually decides whether to keep or drop the packet.

2An inbound datagram fragment close to the length of the interior LAN’s MTU will have to be fragmented again in order toaccommodate the TO LANON header.


8.7 IPsec Authentication, Integrity, and Anti-Replay

In order to resist active attacks, every internal message is stamped with a sequence number and authenti-cated. Operations with improper authentication or sequence numbers are dropped and a warning is issued.Authentication is always the last step performed when preparing internal messages for transmission.

We use IPsec keyed MD5 authenticators [3, 59] with predistributed secret keys; these provide bothauthentication and integrity checking. Keyed MD5 combines a 128-bit secret key with a variable amountof data and produces a 128-bit hash, which is then affixed to the data to be transmitted. The recipientimplicitly concatenates the secret key (which was not transmitted) to the transmitted data and repeats theMD5 computation, verifying that the locally computed hash agrees with the transmitted hash.

IPsec uses the concept of a security parameters index (SPI) [5] to select security transformations. An SPIis a 32-bit integer that both the sender and the recipient agree identifies a sequence of security transforms andthe associated keys that are to be applied to an IP message. In our system, the configuration file specifies aunique SPI for each (operation type, originator native IP address, destination native IP address) tuple. Thusan interior node transmitting a cell to the leader uses a different SPI (and hence different authenticationand/or encryption keys) than another interior node, and both are different from the SPI used by the leader tomulticast TO LANON operations.

Each (operation type, originator IP address, destination IP address) also uses a different sequence num-ber. Sequence numbers are 64 bits long, so they should suffice for quite a long protocol run before wrappingaround (584 years assuming consumption of one billion sequence numbers per second).

8.8 Network Representation

Our protocol header specifications are big-endian for the most part. That is, the most significant bytes ofmulti-byte words appear first on the wire.

We used the Universal Stub Compiler (USC) [65] to convert from native order to protocol order. Sinceour experimental platform was little-endian (80x86-based), the USC-generated code actually did move bytesin the translation process.

Chapter 9

The Trusted Gateway Protocol

9.1 Synopsis

The TG protocol1 uses secret-key encryption to protect cell transmissions from interior nodes to the leader.This is why the protocol is “trusted”—the leader can decrypt each operation and tell exactly what the interiornode is requesting.

9.2 TO GW Operations

We use the IPsec Encapsulating Security Payload (ESP) in transport mode with IDEA-CBC (cipher blockchaining) cryptography [4] to encrypt TO GW operations. The use of IDEA-CBC in ESP is not codifiedin an RFC, but IDEA is well known [49] and uses much larger keys than DES (128 bits versus 56 bits).We followed the layout of fields specified in DES-CBC [58], simply substituting the IDEA for DES. TheCBC initial value (IV) is chosen by a linear congruential generator. The use of the CBC encryption modewithin the keyed MD5 authentication (using a different key) provides integrity and secrecy, while the use ofsequence numbers provides anti-replay protection.

Each TO GW cell is authenticated and encrypted independently. In particular, fragmentation of an ex-ternal packet into cells happens before any cryptographic transformations are applied. We start with theexternal IP packet. This packet is fragmented into 1420-byte chunks and the last chunk is padded out to1420 bytes if necessary. (A chunk is the interior of a cell.) Each chunk is then “grown” into a 1500-bytecell as pictured in Figure 9.1. The fragment identification,offset, and data length fieldsin the Trusted Gateway Header (TGH) of Figure 9.2 are used by the leader to indicate how to reassemblechunks into external IP packets. The fields have the same meanings as they do in IP fragmentation [73].Once the chunks are later reassembled by the leader, the external IP packet appears in precisely its originalform, even if it was itself an IP fragment.

We have already explained the role of the operation,sequence number,fragment identifi-cation, fragment offset, and data length fields of the TGH in Figure 9.2. The version andprotocol fields identify this packet as belonging to the TG protocol. The checksum is straightforward

1The TG protocol was at times called the “local anonymous” and the “synchronous encrypted gateway” protocol. Our imple-mentation retains obsolete monikers such as LAH and SEGW.

86 Chapter 9. The Trusted Gateway Protocol

IP Header

0 8 16 24 31

AuthenticationHeader


EncapsulatingSecurityPayloadHeader

Trusted GatewayHeader

TrustedGatewayHeader

ESP TrailerExternal IP Packet Chunk

(1420 bytes)

8 16 24 310

Figure 9.1: Format of the Trusted Gateway TO GW operation. The scale is in bytes.The Authentication Header protects the light-shaded areas, and thedark-shaded areas are both authenticated and encrypted.

vers proto sequence number

sequence number

0 8 16 24 31

fragment offset data length

checksum

reserved

reserved

reserved

reserved

op

fragment identification

next header cellID

sequence number sequence number

8 16 24 310

Figure 9.2: Format of the Trusted Gateway Header. The operation (TRIGGER,TO GW, etc.) is specified in the op field. The scale is in bits.

Chapter 9. The Trusted Gateway Protocol 87

and arguably unnecessary, since internal messages are always authenticated and integrity-checked. Thenext header field gives the IP protocol number of the enclosed chunk; in our protocol it is always 4,which is IP’s encoding for encapsulated IP packets [81]. Finally, the cellID field is a nonzero 8-bit numberused to identify the cell within this round.

9.2.1 TO GW BLATHER Operations

When an internal node has nothing else to send to the leader, it sends blather. Blather is built from achunk of all zeros just like a TO GW operation, except that the operation field in the TGH is set toTO GW BLATHER.

At one time we used a flag in an ordinary TO GW operation to denote blather. We promoted blather toits own operation because that put it in a different sequence number space, which enabled a performanceimprovement. The improvement is that each internal node keeps a completely prepared blather cell in cacheat all times. When the node is requested to transmit in a round, it sends the cached blather if nothing elsehas been prepared. This spares the delay associated with having to obtain an available sequence number andapply encryption and authentication to a dummy message. Once blather has been transmitted, the node isno longer on the round’s critical path, so it prepares another cached blather cell for the future.

9.2.2 TO GW REPEAT Operations

Since contention at the gateway is possible, sometimes chunks have to be retransmitted. We cannot simplyretransmit fully prepared cells, since this is an obvious sign to an adversary that the chunk was not acceptedin the previous round. Thus chunks that need to be repeated are indepedently promoted into cells.

TO GW REPEAT is a performance improvement in the same vein as TO GW BLATHER. Immediatelyafter an interior node transmits a non-blather chunk, it prepares a TO GW REPEAT version of the samechunk, just in case the chunk is not accepted in the current round. When an interior node is requested totransmit, it sends the TO GW REPEAT if the cell it sent in the previous round was not accepted.

9.2.3 Reacting to TO GW-type Operations

Once the leader of an n-node network has received TO GW operations from each internal node, it collectseach of the n cellIDs (including its own) and accepts one of them:

1. An unusedID is chosen uniformly at random from the set of possible nonzero cellIDs that werenot submitted.

2. Any cellID that was submitted by more than one node is eliminated from consideration, even if itwas a TO GW REPEAT or TO GW BLATHER.

3. Any remaining cellID that is a TO GW BLATHER is eliminated from consideration.

(a) If any submitted cellIDs remain, then the leader selects one of them uniformly at random toaccept. The chunk in the accepted cell is then defragmented; if all chunks are present, the leadertransmits the resultant external IP packet in the ordinary way, subject to IP routing and ARPconsiderations.

(b) Otherwise, the leader “accepts” the unusedID.


Having accepted a cell, the leader must communicate this to the other interior nodes so they can releasetheir resources or try again in the next round. The leader does so by putting the accepted cellID intothe cellID field of the TGH in its next TRIGGER or POLL operation. Note that non-leader interior nodescannot distinguish between cases (a) and (b) above; all they can determine is whether their cell was acceptedor not.

Let a round be called successful if at least one cellID remains after step 2, meaning that not everysubmitted cellID collided with another. A round that is not successful is simply wasted effort. If eachnode’s cellID selection is uniform and independent in the range 1 : : : c, then the probability that a roundis successful is

c � (c� 1)n�1

cn=

�c� 1

c

�n�1

=

�1�

1

c

�n�1

:

Assuming also that cellID selection between rounds is independent, the expected number of rounds be-tween failures is

1

1��1� 1

c

�n�1 :

In our system with n = 12 nodes and c = 255, this comes to only 23 rounds. Clearly, one should consider alarger cellID space. Using 16 bits instead allows 5958 rounds to pass between failures on average, whichis enough for over two minutes of operation on our lightly loaded TG network. Using 32 bits allows formonths of successful operation.

9.3 TRIGGER and POLL Operations

When the leader is ready to start a new round it multicasts a TRIGGER2 operation as shown in Figure 9.3.Only the version, protocol, operation, sequence number, checksum, and cellID fieldsare populated in the TGH shown in Figure 9.3 and 9.2. Each interior node should receive the multicast andrespond with a TO GW-type operation.

Nodes that do not respond “quickly enough” (according to a configuration parameter) are treated to aunicast POLL by the leader. A POLL’s TGH is formed just like a TRIGGER’s; only the operation fielddiffers. The leader continues to POLL quiet nodes until it receives a response. In our experiments, POLLsonly took place when the anonymous network was booting: the leader had started but the interior nodes hadnot yet begun.

9.4 TO LANON Operations

TO LANON operations take place as described in x8.6.4. The version,protocol,operation,sequencenumber, checksum, and next header fields are populated in the TGH shown in 9.2.

2For historical reasons, the TRIGGER operation is actually called REFLECTION in our implementation; it “reflects” the mostrecently accepted cell.


IP Header

0 8 16 24 31



Trusted Gateway Header

8 16 24 310

Figure 9.3: The Trusted Gateway TRIGGER and POLL Operations. The op fieldwithin the TGH determines the operation (Figure 9.2). The scale isin bytes. The shaded area (i.e., the whole packet) is protected by theAuthentication Header.

IP Header

0 8 16 24 31



Trusted Gateway Header

8 16 24 310

External IP Packet(variable size)

Figure 9.4: The Trusted Gateway TO LANONOperation. The scale is in bytes. Theoriginal inbound packet is in the External IP Packet area. OrdinaryIP fragmentation is used if necessary to transmit this operation. Theshaded area (i.e., the entire packet) is protected by the AuthenticationHeader.


0.00

0.20

0.40

0.60

0.80

1.00

1 2 3 4 5 6 7 8 9 10 11 12

xDem

ux C

PU

Con

sum

ptio

n

Number of Nodes in Network

CPU Utilization vs. Network Size

Node 1 (leader)Node 2 (bystander)

Figure 9.5: Trusted Gateway CPU utilization scalability. The CPU consumed byxDemux is averaged over five trials.

9.5 Performance Results

9.5.1 Scalability

CPU utilization. The message of Figure 9.5 is that the leader’s CPU consumption is a scalability bottle-neck in the Trusted Gateway protocol.

Throughput. Figure 9.6 shows throughput in inverse proportion to the number of nodes. This is expected:each additional node adds an essentially constant delay to the round, the delay associated with the leaderreceiving and processing another TO GW packet. Figure 9.6 shows the actual results and predicted resultswhere the additional delay is constant and regular. The parameters for the predicted curve were computedfrom a single arbitrarily chosen round in the trace from case n = 16; the curve is

PredictedThroughput = 0:0172=(0:002312 + 0:00138 � (n� 2)):

In absolute terms, Figure 9.6 shows unimpressive outbound throughput even though it may be acceptablefor some applications. One possible improvement is to use a stream cipher rather than a chaining-modeblock cipher. A stream cipher combines a pseudorandom stream (which may in fact be generated by a blockcipher in a chaining mode) with a plaintext in a very simple way, such as by exclusive-or. The “key” isjust the current index into the pseudorandom stream. The advantage for us would be realized in the leader’sability to precompute the stream segment associated with the expected next transmission and so effectivelyremove the decryption operation from the critical path. As it stands, the leader cannot begin to decrypt atransmission until it arrives in full3. The SS protocol uses a stream cipher and so enjoys a performance

3IPsec does not specify a standard stream cipher, presumably due to a stream cipher’s need for strict synchronization betweensender and receiver combined with the unreliability of IP.


0.00

0.50

1.00

1.50

2.00

2.50

3.00

3.50

4.00

4.50

5.00

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

Thr

ough

put (

Mbi

ts/s

ec)


Throughput Achieved vs Network Size

Actual GatewayPredicted Gateway

Figure 9.6: Trusted Gateway throughput. One node transmits and the other areidle participants. (The single-node case achieves throughput of 39Mbits/sec.)

boost; based on our experience there, we expect that a stream-cipher based TG protocol would be able tosustain outbound throughput of 1 Mbit/sec on 16 nodes of our equipment.

While that would be an improvement, it does not address the O(1=n) trend. Since the leader has to com-municate individually with each node, its memory consumption will also grow proportional to the networksize. Unfortunately, there seems to be little to do with the network to substantially improve scalability thatdoes not require extra trust assumptions. For example, the network could be arranged as a tree of trustedgateways. Although this would improve the scalability of the network, each parent node would be anotherlikely and profitable target for infiltration, for a parent cannot sensibly decide which of its children’s packetsto forward without determining at least which of them are dummy messages.

Delay. Figure 9.7 shows the average delay in msec for an internal node’s chunk to reach the gateway whenthere is no contention. The delay is computed as half of the average time spent processing a round. Thelump and slope change as the 13th node is added correspond to extra switch uplink traversals (see Figure7.1).

9.5.2 Contention Behavior

Contention Throughput. In Figure 9.8 we see a familiar picture. Contention for the gateway looksjust like delay from a fixed client’s point of view, so increased contention results in inversely proportionalthroughput.

In Figure 9.9, the same experimental data is plotted as the network aggregate throughput. This graphmakes it clear that a single TCP client transmitting at maximum speed does not use the full bandwidth ofthe gateway.


0

2

4

6

8

10

12

14

16

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

Ave

rage

Brid

ge A

cces

s D

elay

(m

sec)


Bridge Access Delay vs Network Size

Figure 9.7: Trusted Gateway access delay. One node transmits and the other areidle participants.

0.00

0.10

0.20

0.30

0.40

0.50

0.60

1 2 3 4 5 6 7 8 9 10 11 12

Mea

n T

hrou

ghpu

t per

Clie

nt (

Mbi

ts/s

ec)

Number of Transmitting Nodes in 12-Node Network

Throughput vs. Contention

Figure 9.8: Trusted Gateway contention throughput on a 12-node network. Thereported throughput is the mean throughput achieved by each clientover five trials.


0.00

0.20

0.40

0.60

0.80

1.00

1.20

1.40

1.60

1.80

1 2 3 4 5 6 7 8 9 10 11 12

Agg

rega

te N

etw

ork

Thr

ough

put (

Mbi

ts/s

ec)



Figure 9.9: Trusted Gateway contention throughput on a 12-node network. Thereported throughput is the sum of each transmitting node’s achievedaverage throughput over five trials.

Contention delay. In Figure 9.10, we plot the delays experienced by the 5 clients in a 7-node, 5 transmittercontention experiment. This plot illustrates the gateway access protocol described in x9.2.3, in particular,its fairness toward simultaneous TO GW senders. The five visible mounds correspond to the five trials of theexperiment. At the beginning of each round, each client’s x-kernel reports how long it has been trying tosend a chunk; if it is only sending dummy messages, it reports zero for that round.

9.5.3 Indistinguishability Experiments

The outbound indistinguishability experiments show that the leader is not able to sustain quite as highoutbound throughput when it transmits as the other nodes are. Referring to Figure 9.5, this is not surprising;for a network of 7 nodes, the leader is already spending over 80% of its time in xDemux alone. Not muchtime remains for another process to drive TCP, and ethtap polling happens infrequently.

Figure 9.11 shows how throughput varies with the transmitting node. Differences in achieved throughputbetween nodes 1 through 6 are slight but apparent. We do not know the precise cause of the disparity, but itis worth noting that node 5 is the only Hewlett-Packard PC in the network.

Inbound indistinguishability is a little harder to understand. Recall that throughputs are computed frompacket traces taken in the leader’s x-kernel, not the time that the packets reach the intended node’s TCPlayer. So the only factors controlling the connection peer’s transmission rate are those having to do withTCP flow control. Since we have not modified TCP, each node advertises TCP windows appropriate to whatit sees as an unobstructed network link. As long as the receiving node can keep sending ACKs in a timelyfashion, the peer can transmit at a high rate.

Our implementation allows a single node in a 7-node network to anonymously receive about 14 Mbits/secof TCP data. This is very good news for download-oriented applications such as the WWW protocols: 14Mbits/sec is almost 10 times the capacity of a T1 link.


0

1

2

3

4

5

6

7

8

9

0 1000 2000 3000 4000 5000 6000

Cel

l Acc

epta

nce

Del

ay (

roun

ds)

Time (in rounds)

Cell Queueing Delays -- 7 nodes, 5 transmitters

Node 1Node 2Node 3Node 4Node 5Node 6Node 7

Figure 9.10: Trusted Gateway contention delays on a 7-node network. In the con-tention experiment with 5 simultaneous transmitting clients (nodes 0through 4), each client suffers a delay of about 5 rounds on average.Five sequential trials of the experiment are shown. An unrelated OSevent caused node 5 to transmit briefly during trials 3, 4, and 5.

1.04

1.04

1.05

1.05

1.06

1.06

1.07

1.07

1.08

1.08

1.09

1 2 3 4 5 6 7

Mea

n T

hrou

ghpu

t (M

bits

/sec

)

Transmitting Node

Throughput Achieved vs. Transmitting Node

Figure 9.11: Trusted Gateway outbound indistinguishability on a 7-node network.The leader (node 0) runs slower than the other nodes because its CPUis so heavily utilized. The other nodes experience similar averagethroughputs. The errorbars show the standard deviation of throughputover the 10 trials.


13.80

14.00

14.20

14.40

14.60

14.80

15.00

15.20

1 2 3 4 5 6 7

Mea

n T

hrou

ghpu

t (M

bits

/sec

)

Receiving Node

Throughput Achieved vs. Receiving Node

Figure 9.12: Trusted Gateway inbound indistinguishability on a 7-node network.Nodes 2 through 7 experience similar average throughputs, but node1 uses loopback to its advantage. The errorbars show the standarddeviation of throughput over the 10 trials.

When an inbound packet burst arrives at a node, that node encapsulates each as a TO LANON operationand multicasts it to the network. We employ a loopback mechanism so that the receiving node itself can reactto the TO LANON operation. This gives the leader a substantial advantage over the other nodes when it isreceiving packets, as is evident in Figure 9.12, because our connection peer always transmitted to the leader’snative IP address. We attempted to model the network delay that the other nodes suffer before receiving theTO LANON operation, but the coarse resolution of our thread scheduler (10ms) and our reluctance to busy-wait for this purpose made it impractical.

The lesson that emerges from this investigation is one we already knew: IP anonymity does not automat-ically obscure observable properties of higher-layer protocols. Besides that, the inbound TCP throughputachieved is remarkable: about 14 Mbit/sec or better.

9.5.4 Network Utilization

Network utilization is a strange concept on a switch. If two ports on a 12-port switch are 100% utilized andthe rest are idle, it is misleading to report this as only 2=12 = 17% utilization.

We have chosen to treat our stacked switches as a single large network hub for the sake of reportingnetwork utilization. That is, we assume that the anonymous network is a broadcast network so every NICreceives every transmitted packet. This greatly overestimates the impact of our protocols on a switchednetwork.

To generate the graph in Figure 9.13, we computed the average round duration from the traces, andinferred the number of bits that have to appear on the wire in each round. Therefore, the graph showsonly the overhead associated with the TG protocol. The cost of carrying chunks to the leader is consideredoverhead, since it is indistinguishable from dummy messages. All remaining bandwidth is available for


0.00

0.02

0.04

0.06

0.08

0.10

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

Net

wor

k U

tiliz

atio

n


Network Utilization vs Network Size

Figure 9.13: Trusted Gateway network utilization as the network grows. This isthe baseline utilization, i.e., the overhead incurred by running the TGprotocol with an idle gateway.

inbound communications or competing non-anonymous traffic on the same LAN. The graph shows that theTG protocol consumes less than 8% of the 100 Mbit/sec switch capacity (seen as hub capacity) regardless ofthe network size. As the network grows larger, the CPU-boundedness of the leader creates enough networkidle time to compensate for the increased amount of interior node communication.

Chapter 10

The Superposed Sending Protocol

10.1 Synopsis

The SS protocol differs from the TG protocol primarily in its formulation of TO GW operations; TO LANONprocessing closely matches the description given in Chapter 9. The prima facie motiviation for investigatingSS is its distribution of trust among all of the nodes in an anonymizing network, leaving no particularlyvulnerable nodes. But we also discover some performance advantages.

10.2 Forwarding Topology

We can organize a network of SS nodes into a tree without introducing any additional trust assumptions.This is because the mathematics of the protocol allow a node to superpose (join) two SS TO GW cells withoutknowing whether either is a dummy message or not. When the superposed result is eventually decrypted,the dummy messages are automatically eliminated. So in a tree with leaves, middle nodes and one root,the middle nodes can forward their children’s cells along with their own cells without being able to decrypttheir children’s cells. Since cells do not grow as they climb the tree, collisions are possible and have to bedealt with. We examine these issues more closely in the following sections.

The tree we used in our experiments is depicted in Figure 10.1. This tree closely parallels the physicalswitch topology of Figure 7.1. The leftmost node on each switch is the parent for all of the other nodes onthe switch and the leftmost node of the next lower switch.

10.3 Key-Sharing Graph

Recalling the simplified SS explanation in x3.11, the mechanism that distributes trust in a SS network isthe distribution of multiple keys between nodes. A TO GW operation protected by k different keys cannotbe decrypted until all k keys are applied. In most of our SS experiments, each node protects its TO GWoperations with two keys as depicted in Figure 10.2. This provides substantially more protection than theTG protocol, since infiltration of two nodes is now required to discover the true source of a TO GW. See[18, 67] or especially [96] for good discussions of the anonymity properties of key-sharing graphs.

When a parent has collected all of its children’s TO GWs, it applies its keys to the combined cell andforwards the result upstream.

98 Chapter 10. The Superposed Sending Protocol

1

2 3 4 5 6

7

8 9 10 11 12

13

14 15 16

switch 22 uplink

switc

h 20

upl

ink

Figure 10.1: Superposed Sending forwarding topology. In a network of fewer than16 nodes, the missing nodes are simply omitted from the tree.

Chapter 10. The Superposed Sending Protocol 99

1

2

3

4

n

Figure 10.2: Superposed Sending key-sharing graph. Each edge represents a singlekey shared between the two incident nodes.


Example. Consider a 14-node network and let Kfi;jg be the key shared by nodes i and j. Node 14 applieskeys Kf1;14g and Kf13;14g to its TO GW cell. Node 14 does not transmit its TO GW directly to the leader, butrather to node 13. Node 13 then has all of its children’s cells, so it applies the keys it possesses, Kf13;14gand Kf12;13g. The dual application of Kf13;14g cancels that key from the accumulated cell’s protection,but the cell remains protected by keys Kf1;14g and Kf12;13g. Node 13 sends this cell on to node 7. Aftercombining all of its children’s cells, node 7 will forward the accumulated cell to node 1 protected by Kf6;7gand Kf1;14g; all other intermediate keys will have been applied twice, and so will have canceled. Node 1can then apply Kf1;14g but it still cannot decrypt the result, since it is protected by Kf6;7g. Only when all ofnode 1’s children have contributed do all of the keys cancel each other out. Node 1 is then the first (and only)node to directly recover the decrypted, superposed cell. It can then initiate collision resolution if necessary.

10.4 Key Material and Superposition

The encryptors we apply to cells are actually cell-length pseudorandom sequences whose seeds are theshared keys. We use IDEA in output feedback mode [49] as the generator, hence the protocol labeledofbprsg (output feedback pseudorandom generator) in the x-kernel protocol graph of Figure 7.3. Whenwe refer to “combining” a key, we really are talking about the application of the cell-length pseudorandomsequence corresponding to the key and current state of the generator.

Every key is shared between two nodes1; we designate one of them the positive node, and the other thenegative node. We use addition and subtraction instead of exclusive-or to combine cells and cryptographicmaterial. It does not matter which node is positive with respect to a key, only that the key is added once andsubtracted once within a round. It is a simple administrative matter to keep track of a key’s sign at a node,so we will gloss over it in this text.

When a parent receives a cell from a child, it combines (adds or subtracts) the keys it shares withthat child and then combines (adds) the resulting cell with the parent’s currently accumulated cell. Thecombination of cells is always addition—keys are the only things ever subtracted from cells. So althoughthe keys will cancel each other out by the end of the round (each being added and subtracted once), the cellthat emerges is the sum of all submitted cells.

10.4.1 Arithmetic

Our arithmetic functions, built from gmp (Gnu Multiple Precision [25]) components, treat a cell as a singleunsigned integer represented as a sequence of 356 unsigned 32-bit (longword) big-endian integers with theleast significant longword appearing first on the wire. Overflows during arithmetic operations between cellsare discarded, but carries and borrows within the longwords of a single cell are propagated normally. Soaddition, subtraction, and division retain their usual meaning with respect to this representation.

10.5 TO GW Operations

During a round, each node decides whether to transmit a 1410-byte chunk or not depending on the currentsystem-wide collision resolution state. Assume for the time being that a node decides to transmit. It builds

1This is always true, regardless of the key-sharing graph complexity.


8 16 24 310

0 8 16 24 31

identification fragment offset

data length

Figure 10.3: Fragmentation Header. The scale is in bits.

8 16 24 310

0 8 16 24 31

cellID collisions 0 0

Figure 10.4: Collision Header. The scale is in bits.

IP Header

0 8 16 24 31



External IP Packet Chunk(1424 bytes)

8 16 24 310

Superposed Sending HeaderColsHdr

FragmentHeader

CarryPad

Figure 10.5: Superposed Sending TO GW operation. The scale is in bytes. Thelightly-shaded area is protected by the Authentication Header. Thedarkly-shaded area, the SS payload, is also encrypted by combinationwith keys and possibly other cells.


8 16 24 310

sequence number

0 8 16 24 31

checksum op

stack 0 stack 1accept 0 accept 1

counter0 0

remainder

collision

sequence number

Figure 10.6: The Superposed Sending Header. The scale is in bits.

a chunk, storing the fragmentation information in the fragmentation header seen in Figure 10.3. It appendsone longword of carry padding (all zeros) to the chunk. It then chooses a nonzero cellID at randomand populates the bytes of the collision header (Figure 10.4) with the cellID, 1, 0, and 0 from left toright. The “1” in the collisions field indicates that this cell contains the superposition of only one cell.(The fact that we have only 8 bits allocated to this field means that this protocol only supports anonymizingnetworks of up to 255 nodes.)

The 1424 bytes prepared so far—collision header, fragmentation header, chunk, and carry pad—is calledthe superposed sending payload. Arithmetic between cells and keys always concerns the byte SS payloadalone. The SS payload is shaded darkly in Figure 10.5.

Once prepared, the SS payload is then combined with the corresponding parts of other accumulatedcells (if the node is a parent) and combined with the node’s keys. The operation field in the SSH header(Figure 10.6) is set to TO GW. The other unfamiliar fields in the SSH have to do with collision resolution andwill be described shortly. The remaining fields in the packet (Figure 10.5) are filled in the usual way. Thecompleted cell is now ready for transmission to a parent.

10.5.1 Dummy Messages

To send a dummy message, a node sets the SS payload to zero. The cell is then combined with appropriatekeys and children cells and transmitted as an ordinary TO GW operation. Since the root eventually recoversthe sum of all submitted cells, dummy messages do not interact with other submitted cells and are in a senseautomatically canceled.

Note that the SSH is not encrypted in any SS operation. Whereas the TG protocol encrypted the TGH inorder to hide the distinction between a TO GW operation and a TO GW BLATHER operation, the SS protocolrepresents real cells and dummy cells alike as TO GW operations.


10.6 Collision Resolution

When cells are added, the collisions field in the collision header (Figure 10.4) is coherent—that is, itdoes not absorb overflow from other fields, so the collisions shown in the grand total cell is actuallythe sum of all of the submitted collisions fields. (This assumes that each node is following the rules ofthe protocol, namely, that each cell’s collision header is populated precisely as stated above.)

To see this, note that no overflow is carried into the collision header longword during an addition,since it is the least significant longword in the representation. Within the collision header, the less signif-icant bytes to the right of collisions are zero, so they will not generate carry into the collisions field.Summing cellIDs could well generate carries, but since cellID occupies more significant bytes thancollisions, any generated carry is propagated to the next longword.

When preparing TO GWs, each node sets its collisions field to either 0 or 1 before combining withkeys and other cells. This means that the coherence of the collisions field only holds in a network ofup to 255 nodes (in the case that all nodes simultaneously collide).

When transmitting, nodes add the all-zero carry padding to their chunk in order to provide roomfor carries due to addition of cells’ rightmost (most significant) chunk longword. Our collision resolutionprocedure requires the actual sum of all cells to arrive at the leader. Only one byte of carry space is reallyrequired, given our restriction to networks of size 255 or less.

When the leader recovers the superposed cell for the round, it inspects the resulting cell’s collisionsfield in the collision header. If this field is zero then all submitted cells were dummy messages. If the fieldis one, then exactly one cell was submitted, so there was no collision; the cell is accepted (and emitted if allchunks are present). Otherwise there was a collision.

10.6.1 Strategy

Our collision resolution technique was suggested and described by Pfitzmann [67], who attributed it to jointwork with Klaus Echtle. The technique provides deterministic resolution of k colliding cells in k rounds.This means that the bandwidth consumed to deliver collisions to the leader is not wasted (although most ofthe colliding cells are delayed) and access to the leader’s gateway is fair.

Once the leader has recovered a collided sum S of k > 1 nonzero cells, it initiates resolution. The leadercomputes the mean contributed cell S=k and encloses it in the next round’s TRIGGER operation, signallingthe beginning of a collision resolution sequence of rounds. Nodes that did not contribute to the collisionremain silent during the resolution rounds.

As Pfitzmann [67] notes when comparing this scheme to randomized procedures for collision resolution,

The essential point is that—as long as not all of the cells were equal—there is always at leastone cell less than and one cell greater than S=k. No bandwidth is wasted, because the nodesnever all happen to choose the same coin flip, and so the nodes never all immediately collide orremain silent in the following round. (p. 132)

Upon receiving the mean cell, each colliding node compares the cell c it is trying to transmit with the mean.If c < S=k, then the node transmits its cell in this round; otherwise, it transmits a dummy message.

The result of this round is another sum of cells. If this new sum is itself a collision, the procedurerecurses. Otherwise, a cell is accepted and possibly emitted.


Sometimes two cells are accepted in a resolution round. Consider the case when two nodes’ cells c1 6= c2

collide. In the first resolution round, one of the nodes will transmit its cell, say c1, and the other will keepsilent. At the end of this round, the leader knows both the sum c1 + c2 from the initial collision and the cellc1 from the resolution round, so it can compute c2 = (c1 + c2)� c1 and accept that also.

If on the other hand c1 = c2, then both cells are equal to the mean, and so neither node transmits in theresolution round. Detecting this, the leader concludes that the cells were equal to the mean, and so acceptsboth of them. In general, if more than two identical cells are submitted, then the leader can deduce andemit all of them in a single round, actually achieving throughput greater than 1. However, our protocol doesnot exhibit this behavior in order to keep collision resolution synchronized and predictable. Whenever kcells collide, our protocol takes exactly k rounds to accept those cells even if each cell is identical. (Theoptimization is technically possible in our protocol, but it also concerns an extremely unlikely event, and sowe did not pursue it.)

The collision resolution strategy certainly does use TO GW bandwidth efficiently. However, there is alittle overhead associated with padding to make sure large enough sums can be transmitted, and there is alsoa lot of overhead associated with the transmission of cell means—each mean is as long as an entire cell.This overhead impacts the performance reported in x10.9.2.

Pfitzmann [67] regards this strategy as the “optimal multiple-access method” for superposed sendingnetworks under assumptions appropriate to our environment. Since the only other direct description of ituses the name “tree-like collision resolution with superposed receiving” [96], we settle on calling it thePfitzmann strategy.

10.6.1.1 Accepting Cells and Tracking Progress

The literature on the Superposed Sending technique supposes by default that reliable broadcast is used toinform transmitting nodes whether their cells were accepted or not: the accepted cell in a round is broadcast(as are most internal messages), and each node compares its transmitted cell against the accepted cell.Two research projects [96, 98] specifically addressed the construction of the sufficiently reliable broadcastrequired for superposed sending on an unreliable network.

Pfitzmann [67] and others [96, 98, 14, 56, 15] do not directly state how to detect that a cell was acceptedin the Pfitzmann strategy. But it can be deduced from a close inspection of the technique, and is implied inone of his examples ([67] p. 134), that the cells accepted in every step of the collision resolution procedurecan be deterministically inferred from a tiny amount of reliably distributed progress data. This represents asubstantial performance improvement over cell broadcast and comparison for detecting accepted cells. Wegive details in the rest of this section.

The use of cellIDs to detect acceptance is not sufficient in the SS protocol. The problem is that thechoice of cellIDs can collide, as they can in the TG protocol. But the leader is not able to recognize thatthe cellIDs also collide unless both cells are revealed in the same collision resolution round. For example,in a collision of 6 cells with 2 cells bearing the same cellID 0x2e, it is quite possible for the first 0x2eto be revealed in the first resolution round, and the second 0x2e to be revealed in the last. If both nodeshad decided “my cell has been accepted” at the first 0x2e acceptance, then neither of them would haveparticipated in the later rounds, completely disturbing the resolution procedure.

So a node cannot simply wait for its cellID to appear; it must pay attention to when it appears. Infact, the use of cellIDs is totally optional; nodes infer that their cells are accepted by watching events ona global collision resolution data structure. Assigning cellIDs and naming them in the accept fields of


the SSH helps with system tracing but is not required for correctness.The primary mechanism used to track resolution progress is the (collision resolution) stack. When a

new collision occurs, the leader pushes k—the number of colliding cells—onto the top of the stack. Theleader computes and encloses the mean in its next TRIGGER operation, sets the SSH collision flag,and sets the clock field to the number of colliding cells minus one. The clock decreases by one in eachresolution round until the collision has been completely resolved. The stack is communicated to the otherparticipating nodes in the SSH stack top and stack next fields. In addition, if the computation ofthe collision mean produced a remainder, then the leader sets the remainder flag in the SSH so that thecolliding nodes can correctly compare their own cells (whose “remainders” are always zero) to the mean.

Since the collision resolution state reflected in the stack is not disguised, any observing adversary caneasily determine how many nodes transmit in each round (but of course not which ones). One compensatingfeature of Pfitzmann’s technique is that a single transmitting node can generate collisions all by itself, justby adding up k of its own cells before transmitting the sum as a TO GW operation. The colliding cells will beemitted within k rounds. So in exchange for a reduced throughput due to the resolution procedure, a singlenode can make it seem like multiple nodes are transmitting. To enable this, one must ensure that there issufficient space in the collision resolution header fields for multiple collisions generated by every node. Ourimplementation does not take advantage of this possibility: each node transmits at most one cell in a round.

10.6.2 Leader Specification

The leader’s procedure for accepting cells and controlling collision resolution is outlined in the “leader”function given in Figure 10.7.

While the stack communicated to the internal nodes is a stack containing the number of collisions ateach recursion step, the leader actually maintains a stack of the collided cells themselves (from whichthe number of collisions is easily obtainable). In the leader specification pseudocode, constructions like“R:top:collisions” refer to the collisions field in the collision header of the cell held on the top of thestack R.

Recall that the leader also functions as an internal node; it can transmit cells as well. The “leader” actionsspecified here happen at the beginning of a round, while the leader’s “internal node” actions conceptuallytake place toward the end of a round, like the other nodes. The internal node actions are given in the nextsection.

10.6.3 Internal Node Description

The behavior of internal nodes is more subtle. Before heading directly into the code, we give some examplesof how things need to work.

We examine collison resolution cycles from the perspective of a a single interior node, starting from theinitial collision and ending when all cells have been resolved. In Table 10.1, the node has just transmitteda chunk in the previous round (not shown), and this round resulted in a collision of 4 cells. Since the nodejust transmitted, it is awake (signified by “!”), and it learns of the collision by seeing the collision flagset in a TRIGGER operation. The TRIGGER also sets the clock to 3 (i.e.,4� 1) and sets the new stacktop to 4.

The node has to decide what to do in clock round 3. It compares its cell against the cell mean that wasenclosed with the TRIGGER (not shown), finds that its cell is less than the mean, and transmits its cell at the


procedure leader(Superposed Cell S, Resolution Stack R)initialize outbound SSHupdate()parse()if R is not empty then

SSH.collision := truecompute mean cell in R:topif the mean produced a remainder then SSH.remainder := 1if this is a brand new collision then

SSH:clock := R:top:collisions � 1 // initialize countdown clocktransmit TRIGGER

procedure update // update stack top, clockif R is not empty then // resolution already in progress

SSH:clock := SSH:clock� 1

R:top := R:top� S // infer untransmitted cellsSSH.stack top := S:collisions // transmitted collisions on topSSH.stack next := R:top.collisions // followed by inferred collisionsif R:top:collisions = 1 then

accept(R:top) // inferred cell is collision-freeR.pop()

procedure parse // decide what to do with Scase S:collisions of

0: // no cells submitted in this roundif R is not empty then // non-submitted cells must be equal

num := R:top:collisions

S := R:top=num // compute the real cellaccept(S) // accept one copyR.pop()R.push(S � (num� 1)) // make copiesSSH.stack next := 1 // accept only one copySSH.stack top := S:collisions � (num� 1) // push the others

// else it was all just dummy messages, no collision, wait for next round1: // one cell submitted in this round

accept(S)if R is not empty and R:top:collisions = 1 then

accept(R:top)R.pop()

default:R.push(S)

procedure accept(Collision-free Cell C)add C’s cellID to the next available accept field in SSHemit cell C if all chunks are present

Figure 10.7: Leader Specification Pseudocode


Clock Awake Alert Stack Cmp

3 ! z 4 <

2 ! ! 1 31 z z 2 10 z z 1 1

Table 10.1: One node’s perspective of the collision resolution process

end of round 3.In round 2, it is again anxious to see what happened; it is still awake. Since it just retransmitted in a

resolution round, it is also alert. This means it looks at the new stack contents to see if its cell was accepted asa result of the previous resolution round. The 1 in the stack indicates that one cell was recovered. The boxed1 indicates that this node’s cell was accepted. At this point the node is happy and sleeps (transmitting only

dummy messages) until the end of the resolution cycle.

Rules for Collision Resolution:

� Every resolution round begins with the leader sending a TRIGGER that names the clock value,updates the stack contents, and encloses the cell mean.

� Every round ends with each node either transmitting its cell or transmitting a dummy message.

� Only those nodes that are awake compare their cell against the transmitted mean.

– If a node’s cell is less than the mean, then the cell (re-)transmits its cell at the end of the round.

– All other nodes transmit dummy messages.

� Every node, awake or not, keeps track of the stack.

� The stack top is the leftmost number shown.

� Awake nodes always become alert one round after they awaken—unless they awaken in the last round,in which case they are immediately alert.

� Only those nodes that are alert check to see if their cell was accepted in the previous round.

� Whenever a 1 (denoting an accepted cell) appears in the stack, it appears in one or both of the toptwo places on the stack. The leader accepts these cells and removes them from the stack (possiblyout of order) in the next round. That is, the transmitted mean always refers to the topmost stack cellcontaining more than one collision. This cell is shown in bold face.

– A consequence of this is that the sum of the stack elements in a round is always equal to the sumof those stack elements in the previous round that are not 1.

� The top of the stack always describes cells that were transmitted in the last round, and the next level(if present) describes cells that were inferred in the last round (see Figure 10.7).



3 ! z 4 >

2 ! ! 1 3 <

1 ! ! 2 1 >

0 ! ! 1 1

Table 10.2: Another node’s perspective


4 ! z 5 >

3 ! ! 3 22 z z 2 1 21 ! z 1 1 2 <

0 ! ! 1 1

Table 10.3: A longer collision resolution example

Table 10.2 visits the same resolution example again from another node’s perspective. This node’s round3 comparison with the mean yields >, so it does not transmit its cell. In round 2, it is both awake and alert.Even though it did not transmit its cell, its cell may have been inferred and accepted by the leader, so it mustbe alert to notice this potential event. As it happens, the single transmitted cell at the end of round 3 wasaccepted, not this node’s untransmitted cell. The node compares its cell to the mean transmitted round 2(which describes the collision of 3 cells that was inferred at the end of round 3). Finding it smaller, the nodetransmits its cell. In round 1, the stack indicates that the inferred cell was accepted and the transmitted cellscollided. Therefore, our node remains awake and alert into the last round when its cell is finally accepted.

Let us consider the longer example depicted in Table 10.3, a collision of 5 cells. In round 4, the nodedoes not transmit. In round 3, the node inspects the stack and determines that its cell was not inferred andaccepted by itself, but rather the inferred cell is a collision of 2 nontransmitters. Furthermore, the stack nowhas a collision of 3 cells on top. It will take three rounds (rounds 3 through 1) to resolve that collision, sothe node goes to sleep. It awakens in round 1 and finds its cell < than the mean and transmits; in round 0,its cell is finally accepted.

10.6.3.1 Specification

The complexity of the interior node’s resolution process is in deciding whether to be awake or alert. In thecode of Figure 10.8, we assume that the interior node has transmitted its cell and collided, and that the nodehas a correct view of the data transmitted in recent SSHs: the clock, cells accepted in the last round, andthe stack R. The first sign of a collision is the collision flag being set in a TRIGGER. This causes thecolliding nodes to be awake but not yet alert when they first enter the internal() procedure.

The code refers to two missing functions. The ReactToMean() procedure compares the node’s cellagainst the just-delivered mean; if the node’s cell is smaller, then the node retransmits its cell, otherwise ittransmits a dummy message. ResolutionDone() is called when the node’s cell has been accepted, and theprocedure causes the node to transmit dummy messages until the collision resolution clock has expired.


procedure internal(Resolution Stack R)if I’m awake then

if clock = 0

I’m alert // always alert in last roundTransmittedLast := whether I transmitted my cell in the last collision roundif I’m alert then

if R:next = 1 then // inferred cell was accepted in last roundif TransmittedLast and R:top 6= 1 then

ReactToMean()

Returnelse // either I didn’t transmit and the inferred cell was accepted

ResolutionDone() // or I did and the transmitted cell was acceptedelse // inferred cell was not accepted

if TransmittedLast thenif R:top = 1 then

ResolutionDone() // my cell must have been acceptedelse

ReactToMean()

Returnelse // I didn’t transmit last round

if R:top = 1 // sole transmitted cell was acceptedReactToMean() // mean is based on nontransmitters like meReturn

else // collision on top of stack gets resolved firstSleep until round (clock� (R:top� 1))

else // not alertReactToMean()

Become alert in the next round

Figure 10.8: Internal Node Specification Pseudocode

It may be useful to compare the code against the sample resolutions. Refer back to Table 10.3. In round4, the node is awake but not alert. The node runs ReactToMean(), which causes the node to send only adummy message. In round 3, the node is awake and alert, but TransmittedLast = false. No inferred cellwas accepted, and R:top 6= 1, so the node sleeps until round 3� (3� 1) = 1. In round 1, the node is awakebut not alert. It runs ReactToMean() and transmits its cell. In round 0, the node is awake and alert withTransmittedLast = true. The inferred cell was accepted in the previous round, but so was the transmittedcell. So resolution succeeds.

10.6.4 Collision Resolution Correctness

We begin by restating the procedure outline.

1 In every round, internal nodes either transmit their cells or they do not (i.e., they transmit dummymessages). The leader infers the nontransmitted sum by subtracting from the known total and thetransmitted sum. If either results in a single uncollided cell, then it is accepted. The leader thentransmits the mean value of the top remaining collided sum.


2 An internal node reacts to a transmitted mean (by transmitting or not transmitting) when the node isawake.

3 In the round after reacting to a mean (i.e., when it is alert), a node determines if its cell was acceptedeither because it was the only transmitter or the only nontransmitter in the round.

The key to the correctness of the procedure is in step 2. A node that reacts (transmits or nontransmits) inone round and then goes on to check for acceptance (item 3) in the next round is clearly working correctly.The details concerning whether it was a transmitter or nontransmitter and so how it must interpret the stackcontents are cumbersome but not deep. So it is enough to believe the following:

Lemma 1. Whenever the leader transmits a mean (step 1), exactly those nodes whose cells contributed tothe mean are awake to react (step 2).

To prove the lemma, we use a simple claim.

Claim 1. Suppose that the result of a round is a collision of k � 1 cells. Then it takes exactly k � 1 morerounds to resolve the collision.

Proof. We proceed by strong induction on k.

Basis k = 1. There is no collision. After 0 rounds, the noncollision is trivially resolved.

Induction Step. Assume that the claim holds for all l � k; we demonstrate that it holds for l = k + 1.We start with k + 1 collisions on the stack. Let t be the number of transmitters and n be the number of

nontransmitters after one round has elapsed. Then t+ n = k + 1. There are k + 1 possible choices for thepair (t; n):

(0; k + 1); (1; k); (2; k � 1); : : : ; (k; 1):

The first of these choices, where 0 nodes transmit and k+1 do not, is the special case where all k+1 nodesoriginally transmitted the exact same cell, and the mean is itself this cell. This becomes (k; 1) on the topof the stack (see Figure 10.7); we set (t; n) := (k; 1) to represent this. The other possibilities are pusheddirectly on the stack, e.g., (2; k � 1) results in 2 on the top of the stack and k � 1 right below it.

In both cases the induction hypothesis applies. The collision on the top of the stack is gone after t � 1

more steps, and the collision below it is gone after n � 1 more steps. Since we waited one round to breakk + 1 into pieces, the collision of k + 1 cells is resolved at the end of 1 + (t� 1) + (n� 1) = (k + 1)� 1

more rounds. 2

The lemma follows from the claim. Remember, the goal of collision resolution is to turn a single sum(collision) of k cells into a sequence of k uncollided cells. Since addition is associative and commutative,the order in which we do this does not matter.

Proof of Lemma 1. We use induction on k, the number of colliding cells contributing to the mean, to provethe lemma.

Basis k = 2. With only two colliding cells, both nodes are awake immediately following their transmission,and so both react to the mean.

Induction Step. Suppose the induction hypothesis holds for k � 2; we show it holds for k + 1.


So assume that k + 1 nodes transmitted and collided. In the round that follows, all k + 1 nodes areawake. Reacting to the mean, t � 0 of the nodes transmit and n � 1 do not, where t+ n = k + 1.

If k+1 > n > 1 or k+1 > t > 1, then both t > 1 and t < k+1 hold. The resolution procedure givespriority to resolving the collision of the most recent transmitters. So we can use the induction hypothesistogether with Claim 1 to see that the transmitting nodes are awake at the right times in the ensuing t rounds;meanwhile, also by Claim 1, the nontransmitting cells can sleep for t rounds before their sum’s mean iscomputed.

Otherwise, if t = 0 and n = k + 1, then this is the case of k + 1 identical cells, each equal to the mean.The leader lies and represents this as t = k and n = 1. All of the k + 1 nontransmitting nodes think theyare the sole nontransmitter and believe that the leader accepted their cell, so they sleep for the rest of theresolution cycle. In the next round, the leader will see 0 transmitters and k nontransmitters since all of thenodes are sleeping. The leader repeats the fabrication so that a total of k harmless nontransmitting roundstake place, with each of the nodes reacting appropriately. 2

10.7 TO LANON Operations

TO LANON operations happen almost precisely as in the TG protocol, only some of the relevant fields are indifferent places in the SSH than they are in the TGH. See x8.6.4.

10.8 TRIGGER and POLL Operations

TRIGGER and POLL operations proceed in roughly the same manner as described in x9.3, but they alsocommunicate the collision state and accepted cells in the appropriate SSH fields. In addition, our use of atree topology changes the destination of TRIGGER operations. Rather than multicasting to every node inthe network, the leader’s TRIGGER only multicasts to leaves in the tree. Each child responds by sendinga TO GW to its parent. When all children have then contacted a parent with TO GW operations, the parentcombines its cell and keys and sends the new TO GW to its own parent. So when all goes well, a singleTRIGGER traverses the tree from leaves to the root. If a child does not respond in a timely fashion, thenPOLLs burrow down into the tree in order to request a response.

During collision resolution, things happen a little differently. The leader multicasts its TRIGGER toevery node. Parents wait on the leaves to report as before, but now they have also obtained the collision statedirectly from the leader.

If parent nodes tracked the collision state from their children’s TO GW SSH headers, then they wouldbe vulnerable to active attacks by the children. For instance, a child could pretend that no collision hadoccurred when one in fact had. If the global resolution state after this fabrication is consistent, then theparent chain wasn’t involved, otherwise it was.

While the current strategy of having parents obtain the collision state from the leader prevents childrenfrom mounting an attack, it clearly allows the leader to mount the same type of attack. See x6.2.5 for asuggestion on how to deal with this threat.


0.00

0.20

0.40

0.60

0.80

1.00

1 2 3 4 5 6 7 8 9 10 11 12

xDem

ux C

PU

Con

sum

ptio

n


CPU Utilization vs. Network Size

Node 1 (leader)Node 2 (bystander)

Figure 10.9: Superposed Sending CPU utilization scalability. The CPU consumedby xDemux is averaged over five trials.

10.9 Performance Results

10.9.1 Scalability

CPU utilization. The message of Figure 10.9 is that the leader’s CPU consumption is a scalability bottle-neck in the Superposed Sending protocol also. However, it runs slightly cheaper than the TG protocol (seeFigure 9.5 on page 90) in absolute terms.

In fact, it actually seems to take less time as the network grows larger. This baffling situation is a sideeffect of a performance improvement that reduces network delay and thereby improves throughput2. Assuggested in x9.5.1, the SS protocol uses a stream cipher instead of a block cipher to encrypt (i.e., addrandom streams to) data: the encryption overhead is removed from the critical path and almost completelyprecomputed. So when the leader receives the last TO GW expected in a round, it sums it with a buffercontaining all of the other TO GW contributions and keys, which immediately produces the superposed sum.After the leader reacts and transmits the TRIGGER for the next round, it precomputes the keys it expects touse in the next round and combines them in an empty buffer. All of this happens in the context of the singlexDemux that delivered the last TO GW of the round.

As the network grows, more and more of the leader’s xDemux calls deliver TO GW operations that arenot the last ones expected in the round. For these, the leader simply sums the enclosed cell with its bufferand returns. This type of operation is much cheaper than the one that also produces two cells (2848 bytes)worth of pseudorandom data, and so the xDemux call becomes cheaper on average in larger networks.

Throughput. Figure 10.10 shows throughput for the SS protocol using “tree” and “flat” forwarding topolo-gies. The tree topology is the one shown in Figure 10.1 on page 98, which is based on the physical topology

2This optimization yields a 700 Kbit/sec throughput improvement on a 16-node network.


0.00

0.50

1.00

1.50

2.00

2.50

3.00

3.50

4.00

4.50

5.00

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

Thr

ough

put (

Mbi

ts/s

ec)


Throughput Achieved vs Network Size

Tree SSFlat SS

Figure 10.10: Superposed Sending throughput. One node transmits and the othersare idle participants. The divergence shows Superposed Sending’sability to exploit distributed resources.

of the network shown in Figure 7.1 on page 68. The “flat” forwarding topology simply designates node 1as the leader and all other nodes as leaves; there are no middle nodes. Both forwarding topologies use thesame key-sharing graph shown in Figure 10.2 on page 99.

The flat SS throughput plot can be thought of as “easily achievable” TG throughput. In the scalabilityexperiments, flat SS behaves very similarly to TG. Only one internal node transmits through the gateway,so cell and cellID collisions do not arise. The payload sizes are nearly identical (1424 bytes in SS, 1420in TG). Both protocols simply churn through the cycle of “multicast the TRIGGER”; “receive TO GW fromevery node”. Flat SS still shows a substantial throughput edge over TG, but since all other factors arecomparable, the difference is due to the ability to precompute the stream cipher. So the divergence betweenthe tree and flat SS throughputs can also be thought of as the difference between SS throughput and an“improved TG” throughput.

Our physical topology contains two bottleneck links between switches. The tree forwarding topologyaccounts for this by maximizing the communication local to a switch and minimizing the contention for thebottleneck links. In response to a TRIGGER, only node 13 uses the switch 22 uplink, and only node 7 usesthe switch 20 uplink under the tree topology.

In the flat SS topology and in the TG protocol, internal nodes transmit directly to node 1 (the leader)at will. Nodes 7 through 12 all attempt to send their TO GW to node 1 at roughly the same time. Theseverity of the delay depends on the bottleneck link, but one can count on at least serializing the messages.Our network uses 100 Mbit/sec half-duplex links between switches, but it is easy to imagine a network ofhigh-speed LANs connected by slow links where the penalties would be much less tolerable.

In addition, the CPU utilization plot (Figures 10.9) shows that the leader is saturated even in smallnetworks, but the other internal nodes are not. The tree SS topology allows the network as a whole to benefitfrom the other nodes’ CPU availability by offloading cryptography and coordination efforts to middle nodesin the tree.


Superior Throughput. As a result of these factors, tree SS exhibits scalability that is far superior toflat SS and TG. At 16 nodes, the tree SS achieves about 1.7 times the throughput of flat SS (amounting toabout 730 Kbit/sec.) The benefit of subtrees is clearly visible in Figure 10.10. When node 8 is added to thenetwork, the flat SS throughput suffers, but tree SS hardly notices the change. In our network, this benefit isdue to CPU availability on node 7.

The ability of the SS protocol to use computational and network resources according to their availabilityis a major advantage over the TG protocol. This is in turn due to the fact that dummy messages can beadded to encrypted cells without decrypting them first. A parent need not know anything about its children’stransmissions in order to combine the cells.

Topological Speedup. This flexibility can result in exponential speedup over the TG protocol (orflat SS) when there is little contention, simply because a tree can be traversed in parallel while the direct-to-leader schemes cannot. To see this, let us consider the network and arithmetic overhead only. By discountingthe precomputation of the stream ciphers, the following analysis actually understates the burden on the leaderin the flat SS and TG topology: the number of stream ciphers computed by the leader grows with the networksize in the flat topology, but not in a binary tree topology.

Suppose it takes t CPU seconds for a node to receive a TO GW packet and add it to a buffer containingprecomputed pseudorandom sequences and u seconds to to format and transmit a TO GW cell. Then theleader of an n-node flat network cannot complete a round in fewer than maxft(n � 1); ug seconds, sincethe leader has to serialize the packets3. However, an n-node complete binary tree network can deliver allcontributions to the leader within (2t + u)blog2 nc seconds. Under the reasonable assumption that t and uare independent of n, the claim follows.

Delay. Figure 10.11 shows the average delay in msec for an internal node’s chunk to reach the gatewaywhen there is no contention. The delay is computed as half of the average time spent processing a round.The curve stops increasing when the added node’s grandparent is more heavily utilized than the new node’sparent. For example, when node 14 is added, node 7 (its grandparent) is already a bottleneck; node 7’s delaywhen transmitting to the leader dominates the delay incurred by having node 14 transmit to the relativelyunburdened node 13, which then gets in line to transmit to node 7.

10.9.2 Contention Behavior

Contention throughput. In Figures 10.12 and 10.13 we respectively see the mean throughput and aggre-gate network throughput as contention increases in a 12-node network. Although a single client does notuse the full bandwidth of the gateway, it comes close to the gateway capacity, which appears to be close to3 Mbit/sec.

Cell collision in the SS protocol is intrinsically more expensive than cellID collision in the TG pro-tocol because SS collision resolution always requires transmission of cell means, and these introduce extradelays. On the other hand, although cellID collision in the TG protocol wastes an entire communicationround, it is comparatively rare (and can be made extremely rare with a larger cellID space).

3This statement is obvious for uniprocessor leaders, but even multiprocessor leaders with a fixed number of processors will haveto serialize for large enough n.


0.00

0.50

1.00

1.50

2.00

2.50

3.00

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

Ave

rage

Brid

ge A

cces

s D

elay

(m

sec)


Bridge Access Delay vs Network Size

Figure 10.11: Trusted Gateway access delay. One node transmits and the other areidle participants.

In our experiments, SS still beats TG throughput even when all nodes transmit simultaneously. Webelieve that this is explained by the raw throughput superiority of SS over TG. The SS protocol has CPUtime available where it is most needed to handle collision resolution—in the non-leader internal nodes, eachof which independently react to and track the collision state. In contrast, the TG network is completelybounded by the leader’s CPU availability. So even though SS contention really is more expensive than TGcontention, SS has the horsepower to handle the extra load.

Instead of transmitting cell means in the SS protocol, Pfitzmann suggests transmitting order-of-magnitudeestimate of the mean and using random choices to recover from ambiguities ([67], p. 135). This would cer-tainly boost contention throughput even further.

Contention delay. In Figure 10.14, we plot the delays experienced by the 5 clients in a 7-node, 5 trans-mitter contention experiment. The five visible mounds correspond to the five trials of the experiment. At thebeginning of each round, each client’s x-kernel reports how long it has been trying to send a chunk; if it isonly sending dummy messages, it reports zero for that round.

This plot illustrates the deterministic nature of collision resolution in the SS protocol. During periodsof simultaneous transmission, collisions of 5 cells cause a delay of 5 cycles with small deviation. Comparethis to the TG image in Figure 9.10. (The 5 transmitting processes are not perfectly synchronized; theysimply start at approximately the same time and run the same code. So while simultaneous transmissionsare expected, they are not guaranteed.)

10.9.3 Indistinguishability Experiments

The basic conclusion to be drawn from the indistinguishability experiments is the same in SS as it is in TG:IP anonymity does not effectively disguise surface characteristics of the TCP level. The experiment resultsare shown in Figures 10.15 and 10.16.


0.20

0.40

0.60

0.80

1.00

1.20

1.40

1.60

1.80

2.00

2.20

1 2 3 4 5 6 7 8 9 10 11 12

Mea

n T

hrou

ghpu

t per

Clie

nt (

Mbi

ts/s

ec)



Figure 10.12: Superposed Sending contention throughput on a 12-node network.The reported throughput is the mean throughput achieved by eachclient over five trials.

0.00

0.50

1.00

1.50

2.00

2.50

3.00

1 2 3 4 5 6 7 8 9 10 11 12

Agg

rega

te N

etw

ork

Thr

ough

put (

Mbi

ts/s

ec)



Figure 10.13: Superposed Sending contention throughput on a 12-node network.The reported throughput is the sum of each transmitting node’sachieved average throughput over five trials.


0

1

2

3

4

5

6

7

8

9

10

0 1000 2000 3000 4000 5000 6000 7000

Cel

l Acc

epta

nce

Del

ay (

roun

ds)

Time (in rounds)

Cell Queueing Delays -- 7 nodes, 5 transmitters

Node 1Node 2Node 3Node 4Node 5Node 6Node 7

Figure 10.14: Superpose Sending contention delays on a 7-node network. Inthe contention experiment with 5 simultaneous transmitting clients(nodes 0 through 4), each client suffers a delay of about 5 rounds onaverage with small deviation. Five sequential trials of the experimentare shown.

Again, we see that the leader achieves noticeably different throughput than the other nodes, but thistime its inbound throughput is worse than the others. Compare this with the TG Figure 9.12 showing betterinbound throughput to the leader. This disparity may be due to the fact that in our 7-node network, morethan two rounds worth of SS communication can take place within a single 10 msec timeslice but one roundof TG communication already exceeds the 10 msec timeslice. So the TG protocol allows the TCP receivingprocess (also running on the leader node) to run more frequently than the SS protocol does, and this worsensthe SS leader’s inbound throughput.

The other nodes seem to handle inbound TCP throughput of over 20 Mbit/sec in the SS protocol. Seex9.5.3 for a caveat regarding this phenomenal figure.

10.9.4 Response Time

As mentioned in x6.2, our system architecture reflects a node’s CPU load in the timing of its response to aTRIGGER. Figures 10.17 and 10.18 illustrate this. In the first figure, node 2 transmits through the gateway10 times and the others are idle. In the second figure, node 5 transmits through the gateway while the othersare idle. In both cases we see that the transmitting node’s response time rises sharply during transmission,while the other nodes are much more subtly affected.

We view this as a relatively minor vulnerability, because it is a generally unwarranted assumption thatany CPU load is evidence of intentional network communication—perhaps the computer is actually com-puting something. Still, by giving the anonymizing layer very high CPU priority, this vulnerability can bereduced. This was not practical in our implementation because of ethtap’s polling of system buffers; ifthat were given high priority, then other jobs in the system would never run.


2.05

2.10

2.15

2.20

2.25

2.30

1 2 3 4 5 6 7

Mea

n T

hrou

ghpu

t (M

bits

/sec

)

Transmitting Node

Throughput Achieved vs. Transmitting Node

Figure 10.15: Superposed Sending outbound indistinguishability on a 7-node net-work. The errorbars show the standard deviation of throughput overthe 10 trials.

18.50

19.00

19.50

20.00

20.50

21.00

1 2 3 4 5 6 7

Mea

n T

hrou

ghpu

t (M

bits

/sec

)

Receiving Node

Throughput Achieved vs. Receiving Node

Figure 10.16: Superposed Sending inbound indistinguishability on a 7-node net-work. The errorbars show the standard deviation of throughput overthe 10 trials.


1.00

1.50

2.00

2.50

3.00

3.50

4.00

4.50

5.00

20 25 30 35 40 45

TR

IGG

ER

res

pons

e tim

e (m

sec)

Time (in sec, resolution 0.5 sec)

Response time of coalition participants during test with transmitting node 2

Node 2Node 5Node 6

Trial beginTrial end

Figure 10.17: Node 2’s response time. We have plotted the response times of 3nodes in the 7-node indistinguishability experiment where node 2transmits. The response time is an average taken every 0.5 seconds.Node 2’s increased response time is clearly visible during its trans-mission cycles.

1.00

1.50

2.00

2.50

3.00

3.50

4.00

4.50

5.00

20 25 30 35 40 45

TR

IGG

ER

res

pons

e tim

e (m

sec)

Time (in sec, resolution 0.5 sec)

Response time of coalition participants during test with transmitting node 5

Node 2Node 5Node 6

Trial beginTrial end

Figure 10.18: Node 5’s response time. We have plotted the response times of 3nodes in the 7-node indistinguishability experiment where node 5transmits. The response time is an average taken every 0.5 seconds.Node 5’s increased response time is clearly visible during its trans-mission cycles.


0.000.020.040.060.080.100.120.140.160.180.200.220.240.260.280.300.32

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

Net

wor

k U

tiliz

atio

n


Network Utilization vs Network Size

Figure 10.19: Superposed sending network utilization as the network grows. Thisis the baseline utilization, i.e., the overhead incurred by running theSS protocol with an idle gateway.

10.9.5 Network Utilization

Since our SS implementation runs so much faster than our TG implementation, it is able to process morerounds the same amount of time. Consequently, it consumes more of the network bandwidth; this tradeoff isclear enough. Figure 10.19 shows our estimation of network utilization with an idle SS gateway. We modelthe network switch as a large hub, as we did in x9.5.4. That makes the graph particularly misleading, sincethe tree SS topology localizes TO GW traffic communication within a switch. In other words, the impact ofadding nodes 8 through 16 is extremely slight on the top switch, but our “virtual hub” model makes it seemexpensive. It might be reasonable to interpret the graph’s message as “16% utilization on each involved7-node switch.”

10.9.6 Key-Sharing Graph Complexity

Our experiments all ran with the simple key-sharing topology of Figure 10.2, in which each node shareskeys with two neighbors. We also investigated the impact of using more complex key-sharing graphs. Morecomplex graphs are more secure, since adversaries need to discover (perhaps by infiltration) correspondinglymore keys in order to deduce the source of communications. As expected, increasing the graph connectivity(number of neighbors per node) increases delay and decreases throughput in a straightforward manner.

Figure 10.20 shows the outbound throughput achieved on a 16-node network with a single transmitter asthe number of neighbors per node increases. (Although from a security perspective it is only useful to sharekeys with 15 neighbors in a 16-node network, we computed and applied 16 shared keys per node to generatethe final point on the graph; each node rather uselessly shared a key with itself.) Even with 10 key-sharingneighbors, a node is able to sustain 1 Mbit/sec outbound throughput in this network.


0.00

0.20

0.40

0.60

0.80

1.00

1.20

1.40

1.60

1.80

2.00

2 4 6 8 10 12 14 16

Thr

ough

put (

Mbi

t/sec

), m

ean

of 1

0 tr

ials

Number of Neighbors Per Node

Outbound Throughput vs Key-sharing Graph Complexity on 16-node Network

Figure 10.20: Superposed sending key-sharing graph complexity. One node trans-mits and the others are idle participants. As the number of sharedkeys grows, so does cryptographic effort and delay.

Chapter 11

Conclusion

11.1 Comparison of the Protocols

The SS protocol meets or outperforms the TG protocol in almost every way, even if one improves the TGprotocol by adopting stream ciphers and cryptographic precomputation (x9.5.1, x10.9.1).

� SS is fundamentally a distributed protocol and is able to take advantage of distributed resources(x10.9.1). A node with excess CPU power or an uncongested network link can be made the parent ofother nodes. This decreases delay and so the whole anonymizing network benefits.

� The SS protocol easily adapts to differing trust requirements. In other words, the performance impli-cations of expanding the key-sharing graph are straightforward (x10.9.6).

� Both protocols require roughly the same amount of random material. In an n-node network, the SSprotocol uses n + 1 128-bit IDEA keys to protect transmissions of 1438 bytes, and the TG protocoluses n 128-bit IDEA keys to protect transmissions of 1448 bytes.

� Although collision resolution is expensive in SS because of the transmission of cell means, it can bemade much cheaper with order-of-magnitude estimates combined with coin flips ([67], p. 135). Agood implementation should be able to guarantee small and bounded delay during contention. Butit will never completely beat TG, since TG never has to transmit anything extra in order to resolve acellID collision.

The TG protocol and all other direct-to-leader schemes are unscalable if one wants to minimize the num-ber of required trusted components. Since transmissions from scrutinized nodes to the leader are encrypted,it is not possible for other nodes to combine intermediate results. The leader has to process every message,which quickly leads to CPU or network link overload.

The primary disadvantage of the SS technique is that the sending process is inherently synchronous.Each configured sender must contribute in a round for the keys to cancel out properly. If a node is unre-sponsive and must be excluded, then the current round has to be abandoned and a new key-sharing graphgenerated and distributed.

The fact that both systems are CPU-bound implies that the performance achieved in our implementationis fragile. “Idle” systems in the experiments are actually quite busy supporting the anonymous network.

124 Chapter 11. Conclusion

Since the x-kernel competes fairly with other processes in our implementation, a single busy CPU canadversely impact the entire network. However, this is a property of our implementation architecture andnot the protocols themselves. Realizing the protocols in a high-priority scheduling domain—as is generallyalready done with implementations of the IP suite—helps make the network performance independent ofuser CPU load on the participant nodes.

11.2 Future Directions

The performance results from this implementation suggest that a high-speed local anonymous network usingca. 1998 off-the-shelf hardware is feasible. Clearly, the shortcomings of our implementation listed in x6.2need to be overcome if these protocols are to be realized in production form. In addition, the followingissues stand out:

� In x10.9.1, we gave evidence that a SS network can be designed so that it puts a heavier share ofthe aggregate CPU burden on inherently faster nodes. Since the network topology can be changedwithout changing trust assumptions, a natural next step is to design the protocols to dynamicallyadapt the network topology according to CPU availability. This would not only improve the network’sperformance under virtually all conditions, but it would also make network auto-configuring.

� The anonymizing network parameters should to be communicated to and controlled by users andnetwork managers in a sensible matter (perhaps by SNMP [83, 89]).

� In x2.3.1, we mentioned an example where a collection of addresses (a secret committee) was theprotected identity [9]. We are not aware of any work that investigates this idea throughly in the contextof anonymity, although secret-sharing schemes [84, 47, 82, 57] and information dispersal algorithms[78, 8] are probably relevant to such a construction.

� In the spirit of minimizing overhead, the protocols can be made to operate on demand, similar to idle“spin-down” response to idleness in CD-ROMs or hard drives. If the gateway has been idle for a while,then the frequency of rounds can be lowered significantly and returned to high-throughput levels whenthe gateway is actually used in a round. One must be careful not to lower the frequency too much,however, since this would impose a possibly unacceptable delay on interior nodes that do want totransmit. Our 16-node experimental SS network ran at about 275 rounds per second; by dropping thisto 10 rounds per second after a few minutes of idle time, CPU and network bandwidth are returned toother uses while causing only a 0:1 second delay to the first node that decided to transmit.

The ability to infer idleness by noticing that rounds are infrequent does not degrade anonymity in theSS protocol, since SS already indicates this by explicitly announcing how many cells are accepted ineach round. However, the TG protocol does hide this information, so the suggested technique woulddegrade its anonymity somewhat.

� The provided IP anonymity should be more fully extended to higher layer protocols. The managementof TCP and UDP ports is critical, both for proper operation and for providing unlinkability betweensubsequent connections. We suggest that anonymous TCP and UDP access be provided to applicationsthat explicitly request anonymous sockets (or another networking abstraction, see [91]).

Chapter 11. Conclusion 125

This represents a possibility that may be quite practical. Rather than shuttling all IP packets throughan anonymizing protocol, a network could maintain a local anonymous channel in addition to itsordinary networking layers. Only those applications that request anonymous sockets would use theslower anonymous network, and so users could directly choose the appropriate time/security tradeoffs.

This last point recalls the building-block nature of our approach. Previous systems have anonymized byproxy, and the communicating applications are kept unaware that the anonymizing mechanisms are in use.In fact, our implementation also currently anonymizes by proxy: applications and even the TCP and UDPlayers are not aware of the anonymization. But by constructing anonymity from the bottom up, we enablethe construction of higher level anonymous interactions.

11.3 Summary

The work reported in this thesis is a step towards the possibility of low-level network anonymization. Al-though our performance results stem from an unusual implementation architecture, we find them encourag-ing. In particular, we have learned that the flexibility of the Superposed Sending technique is a tremendousadvantage in heterogeneous networks. We hope that this arouses interest among other researchers and im-plementors.

The protocols we developed are instances of local anonymity. Local anonymous networks are signifi-cantly easier to establish and maintain than wide-ranging dispersed ones, and we now know that they canperform acceptably without requiring large equipment investments. In addition, we hope that the combi-nation of inconspicuousness with relatively small adversarial search spaces is able to provide a meaningfullevel of protection without significantly contributing to the problem of abuse via abject untraceability.

Bibliography

Internet RFCs are published by the USC Information Sciences Institute at Marina Del Rey, CA, and can beobtained from http://www.isi.edu/publications.html.

[1] Anonymizer FAQ, 1997. See http://www.anonymizer.com/faq.html.

[2] Terms and conditions of the Anonymizer service, 1997. See http://www.anonymizer.com/terms.html.

[3] R. Atkinson. IP authentication header. RFC 1826, 1995.

[4] R. Atkinson. IP encapsulating security payload (ESP). RFC 1827, 1995.

[5] R. Atkinson. Security architecture for the Internet Protocol. RFC 1825, 1995.

[6] Paul Barford and Mark Crovella. Generating representative workloads for network and server per-formance evaluation. In Proceedings of ACM SIGMETRICS ’98, pages 151–160, Madison, WI, June1998.

[7] T. Berners-Lee, R. Fielding, and H. Nielsen. Hypertext transfer protocol — HTTP/1.0. RFC 1945,1996.

[8] Azer Bestavros. AIDA: A bandwidth allocation strategy for distributed time-critical systems. In Pro-ceedings of the First IEEE IPPS Workshop on Parallel and Distributed Real-Time Systems, NewportBeach, CA, April 1993.

[9] Azer Bestavros. Personal communication, December, 1998.

[10] Azer Bestavros, Mark Crovella, Jun Liu, and David Martin. Distributed packet rewriting and itsapplication to scalable server architectures. In Proceedings of the 1998 International Conference onNetwork Protocols. IEEE, November 1998.

[11] Hakim Bey. T.A.Z.: The Temporary Autonomous Zone, Ontological Anarchy, Poetic Terrorism. Au-tonomedia, New York, 1991.

[12] Daniel Bleichenbacher, Eran Gabber, Phillip Gibbons, Yossi Matias, and Alain Mayer. On personal-ized yet anonymous intercation. Technical report, Bell Laboratories, April 1997.

128

[13] Daniel Bleichenbacher, Eran Gabber, Phillip Gibbons, Yossi Matias, and Alain Mayer. On secure andpseudonymous client-relationships with multiple servers. Technical report, Bell Laboratories, April1998.

[14] J. Bos and B. den Boer. Detection of disrupters in the DC protocol. In Lecture Notes in ComputerScience 434 (Eurocrypt ’89). Springer-Verlag, 1989.

[15] Manfred Bottger. Realisierung des DC-Netz-Versuchs und einer einheitlichen Praktikums-Netzschnittstelle (Realization of a DC-Network and Uniform Network Interface Including Lab Expe-riments). Diplomarbeit, Universitat Karlsruhe, Institut fur Rechnerentwurf und Fehlertoleranz, 1990.In German.

[16] Dave Butenhof. Programming with POSIX Threads. Addison Wesley, 1997.

[17] David Chaum. Untraceable electronic mail, return addresses, and digital pseudonyms. Communica-tions of the ACM, 24(2), February 1981.

[18] David Chaum. The dining cryptographers problem: Unconditional sender and recipient untraceabil-ity. Journal of Cryptology, 1:65–75, 1988.

[19] D. E. Comer. Internetworking with TCP/IP, volume 1. Prentice Hall, 3rd edition, 1995.

[20] The Standard Performance Evaluation Corporation. Specweb96. See http://www.specbench.org/org/web96/.

[21] Lance Cottrell. Mixmaster and remailer attacks. See http://obscura.com/�loki/remailer-essay.html.

[22] Lance Cottrell. Mixmaster FAQ. See http://www.obscura.com/�loki/remailer/mixmaster-faq.html, 1996.

[23] Crowds home page. See http://www.research.att.com/projects/crowds. See also[80].

[24] Wei Dai. Pipenet. post to the cypherpunks mailing list, February 1995. See http://www.eskimo.com/�weidai.

[25] TMG Datakonsult. Gnu multiple precision arithmetic library. See ftp://prep.ai.mit.edu/pub/gnu/gmp-2.0.tar.gz.

[26] T.c.X DataKonsultAB. Mysql pthreads. Contained in source distribution at http://www.tcx.se.

[27] Simon Davies. Connected: They are eavesdropping on our every word. The Daily Telegraph, De-cember 16 1997. p. 4.

[28] Simon Davies. Connected: EU simmers over Menwith listening post. The Daily Telegraph, July 161998. p. 6.

[29] Thomas Demuth and Andreas Rieke. Janus web site. See http://janus.fernuni-hagen.de.

129

[30] Thomas Demuth and Andreas Rieke. Anonym im World Wide Web? Datenschutz und Datensicher-heit, 22, November 1998. In German.

[31] W. Diffie and M. E. Hellman. New directions in cryptography. IEEE Transactions on InformationTheory, 22:644–654, 1976.

[32] Shlomi Dolev and Rafail Ostrovsky. Efficient anonymous multicast and reception. In Advances inCryptology: Proceedings of CRYPTO ’97, 1997.

[33] Andy Dustman. EFGA reliable remailer lists. See http://anon.efga.org/�rlist.

[34] Andreas Fasbender, Dogan Kesdogan, and Olaf Kubitz. Analysis of security and privacy in mobileIP. In 4th International Conference on Telecommunication Systems Modeling & Analysis, Nashville,USA, March 1996.

[35] Andreas Fasbender, Dogan Kesdogan, and Olaf Kubitz. Variable and scalable security: Protection oflocation information in mobile IP. In Mobile IP, 46th IEEE Vehicular Technology Society Conference,Atlanta, March 1996.

[36] R. Fielding, J. Gettys, J. Mogul, H. Frystyk, and T. Berners-Lee. Hypertext transfer protocol —HTTP/1.1. RFC 2068, 1997.

[37] Eran Gabber, Phillip Gibbons, Yossi Matias, and Alain Mayer. How to make personalized webbrowsing simple, secure, and anonymous. Technical report, Bell Laboratories, May 1997. See alsoFinancial Cryptography ’97.

[38] Ian Goldberg and David Wagner. Taz servers and the rewebber network. See http://www.cs.berkeley.edu/�daw/cs268/taz.ps.

[39] Ian Goldberg, David Wagner, and Eric Brewer. Privacy-enhancing technologies for the internet. InIEEE Compcon, 1997.

[40] David Goldschlag, Michael Reed, and Paul Syverson. Hiding routing information. Workshop onInformation Hiding, May 1996. Cambridge, UK.

[41] Ceki Gulcu and Gene Tsudik. Mixing email with Babel. In Proceedings of the 1996 Internet SocietySymposium on Network and Distributed System Security, February 1996.

[42] Johan Helsingus, February 20 1995. Press Release.

[43] N. C. Hutchinson and L. L. Peterson. The x-kernel: An architecture for implementing networkprotocols. IEEE Transactions on Software Engineering, 17(1):64–76, January 1991.

[44] IEEE. Carrier Sense Multiple Access with Collision Detection (CSMA/CD) Access Method and Phys-ical Layer Specifications., ANSI/IEEE Standard 802.3 edition, 1985.

[45] Mason Katz. Linux x-kernel IPSEC. Includes ETHTAP description. See http://www.cs.arizona.edu/security/hpcc-blue/linux.html.

130

[46] Charlie Kaufman, Radia Perlman, and Mike Speciner. Network Security: Private Communication ina Public World. Prentice-Hall, 1995.

[47] S. C. Kothari. Generalized linear threshold scheme. In Advances in Cryptology: Proceedings ofCRYPTO ’84. Springer-Verlag, 1984.

[48] Howard Kurtz. Rep. Barr’s Divorce Answers Draw Flynt Fire. The Washington Post, January 121999. p. A7.

[49] X. Lai. On the design and security of block ciphers. In J.L. Massey, editor, ETH Series in InformationProcessing. Hartung-Gorre Verlag Konstanz, Technische Hochschule (Zurich), 1992.

[50] Gia Lee. Addressing anonymous messages in cyberspace. Journal of Computer Mediated Communi-cation, 2(1), 1996.

[51] Raph Levien. Remailer list. See http://www.cs.berkeley.edu/�raph/remailer-list.html.

[52] Bill Lewis and Daniel J. Berg. Multithreaded Programming with Pthreads. Prentice-Hall, 1998.

[53] Steve Lipsher. Vail probe dialing down. The Denver Post, November 23, 1998.

[54] Lucent personal web assistant web page. See http://www.lpwa.com.

[55] Niccolo Machiavelli. The prince, ca. 1515.

[56] Eckhard Marchel. Leistungsbewertung von uberlagendem Empfangen bei Merfachzugriffsverfahrenmittels Kollisionsauflosung (Performance evaluation of superposed receiving with multiple-accesschannel collision resolution). Diplomarbeit, Universitat Karlsruhe, Institut fur Rechnerentwurf undFehlertoleranz, 1988. In German.

[57] Alfred Menezes, Paul van Oorschot, and Scott Vanstone. Handbook of Applied Cryptography. CRCPress, 1996.

[58] P. Metzger, P. Karn, and W. Simpson. The ESP DES-CBC transform. RFC 1829, 1995.

[59] P. Metzger and W. Simpson. IP authentication using keyed MD5. RFC 1828, 1995.

[60] Walter S. Mossberg. Accountability is key to democracy in the on-line world. The Wall Street Journal,January 26 1995.

[61] Ron Newman. The Church of Scientology vs. the Net, 1997. See http://www2.thecia.net/users/rnewman/scientology/home.html.

[62] Arnold Niedermaier. Bewertung von Zuverlassigkeit und Senderanonymitat einer fehlertolerantenKommunikationsstruktur (Evaluation of reliability and sender anonymity in an error-tolerant com-munication structure). Diplomarbeit, Universitat Karlsruhe, Institut fur Rechnerentwurf und Fehler-toleranz, 1998. In German.

131

[63] Computer jokes and threats ignite debate on anonymity. The New York Times, December 31 1994.pp. 1,53.

[64] Office of the Independent Counsel. Referral to the Unied States House of Representatives pursuantto Title 28, United States Code, x595(c), September 9, 1998. A.k.a. the Starr Report.

[65] Sean O’Malley, Todd Proebsting, and Allen Brady Montz. USC: A universal stub compiler. InProceedings of the SIGCOMM ’94 Symposium, August 1994.

[66] Andreas Ort. Implementierung eines MIX-Netzes sowie Konzeption und Realisierung des MIX-Netz-Versuchs (Implementation of a MIX-network and design and realization of MIX-network labexperiments). Diplomarbeit, Universitat Karlsruhe, Institut fur Rechnerentwurf und Fehlertoleranz,1991. In German.

[67] Andreas Pfitzmann. Diensteintegrierende Kommunikationsnetze mit teilnehmer-uberprufbarem Da-tenschutz (ISDN Networks with Member-Verifiable Data Privacy), volume 234 of Informatik-Fachberichte. Springer-Verlag, 1990. Also Ph.D. Thesis. In German.

[68] Andreas Pfitzmann and Michael Waidner. Networks without user observability—design options.Computers & Security, 6(2):158–166, 1987. See also [88].

[69] D. Plummer. Ethernet address resoluton protocol: Or converting network protocol addresses to 48 bitethernet address for transmission on ethernet hardware. RFC 826, 1982.

[70] IEEE/ANSI Standard 1003.1 Information Technology—Portable Operating System Interface(POSIX)–part 1: System Application: Program Interface (API), 1996. See www.ieee.org.

[71] J. Postel. User datagram protocol. RFC 768, 1980.

[72] J. Postel. Internet control message protocol. RFC 792, 1981.

[73] J. Postel. Internet protocol. RFC 791, 1981.

[74] J. Postel. Transmission control protocol. RFC 793, 1981.

[75] Onion routing visualization [traffic analysis threat]. See http://www.onion-router.net/Vis.html.

[76] Privacy and anonymity on the internet. See http://www.onion-router.net.

[77] Christopher Provenzano. Unix pthreads implementation, 1996. See http://www.mit.edu/people/proven/pthreads.html.

[78] Michael O. Rabin. Efficient dispersal of information for security, load balancing, and fault tolerance.Journal of the ACM, 36(2):335–348, April 1989.

[79] Red Hat Linux. See http://www.redhat.com.

[80] Michael Reiter and Aviel Rubin. Crowds: Anonymity for web transactions. Technical report, DI-MACS, 1997. See also [23].

132

[81] J. Reynolds and J. Postel. Assigned numbers. RFC 1700, 1994.

[82] Bruce Schneier. Applied Cryptography. John Wiley and Sons, 2nd edition, 1996.

[83] M. Schoffstall, M. Fedor, J. Davin, and J. Case. A simple network management protocol (SNMP).RFC 1157, 1990.

[84] A. Shamir. How to share a secret. Communications of the ACM, 24(11), November 1979.

[85] C. E. Shannon. Communication theory of secrecy systems. Bell System Technical Journal, 28:656–715, 1949.

[86] Tim Shepard. TCP Packet Trace Analysis. Master’s thesis, M.I.T., 1990. See ftp://ftp.lcs.mit.edu/pub/lcs-pubs/tr.outbox/MIT-LCS-TR-494.ps.gz.

[87] Martha S. Siegel. Anarchy, chaos on the Internet must end. The San Francisco Chronicle, January 21995. Editorial.

[88] Sirene (security in computer networks) publications links. See http://www.semper.org/sirene/lit/sirene.lit.html.

[89] W. Stallings. SNMP, SNMPv2, and RMON: Network Management. Addison-Wesley, 1996.

[90] W. Richard Stevens. TCP/IP Illustrated. Addison-Wesley, 1994.

[91] W. Richard Stevens. UNIX Network Programming. Prentice Hall, 2 edition, 1998.

[92] Paul A. Strassmann and William Marlow. Risk-free access into the global information infrastructurevia anonymous re-mailers. In Symposium on the Global Information Infrastructure, Cambridge, MA,January 28–30 1996. See http://www.strassmann.com/pubs/anon-remail.html.

[93] Paul Syverson, David Goldschlag, and Michael Reed. Anonymous connections and onion routing. InProceedings of the 1997 IEEE Sympsium on Security and Privacy, May 1997.

[94] Andrew Tanenbaum. Computer Networks. Prentice-Hall, 3rd edition, 1996.

[95] David Wagner. Eagle (the President) and the Eagle Beagle [Pager intercepts]. RISKS Digest 19.39.See http://www.CSL.sri.com/risksinfo.html.

[96] Michael Waidner. Unconditional sender and recipient untraceability in spite of active attacks. InEurocrypt ’89, volume Lecture Notes in Computer Science of 434, pages 302–319. Springer-Verlag,1989. See also [97, 88].

[97] Michael Waidner and Birgit Pfitzmann. Unconditional sender and recipient untraceability in spite ofactive attacks—some remarks. Technical report, Fakultat fur Informatik, Universitat Karlsruhe, 1989.See also [88].

[98] Michael Waidner and Birgit Pfitzmann. The dining cryptographers in the disco: Unconditional senderand recipient untraceability with computationally secure serviceability. In Eurocrypt ’89, volumeLecture Notes in Computer Science of 434. Springer-Verlag, 1990. See also [88].

133

[99] P. R. Zimmerman. The Official PGP User’s Guide. MIT Press, Boston, 1995.

[100] Freedom internet identity management system. Zero Knowledge Systems, Inc. See http://www.zks.net/products/whitepapers.asp.

boston university graduate school of arts and sciences

Documents