Backtracking Algorithmic Complexity Attacks Against a NIDS

Randy Smith, Cristian Estan, Somesh Jha

University of Wisconsin–Madison

Algorithmic Complexity Attacks

Vulnerable algorithm: algorithm whose worst case differs from typical case. The larger the difference, the more vulnerable the algorithm.

Examples:

Algorithm Average Worst

Quicksort O(n log n) O(n2)

Hash lookup constant O(n)

Algorithmic Complexity Attacks

Algorithmic Complexity Attack – an attacker induces worst-case behavior in a vulnerable algorithm. Common observable effect is denial of service.

Crosby and Wallach: induced worst-case behavior in hash function implementations.

“Algorithms are now part of the attack surface” (Crosby and Wallach, 2003)

Are NIDS vulnerable? NIDS and IPS are ubiquitous, but…

Do they contain vulnerable algorithms? Can they be exploited?

YES! Only need 1 packet every 3 seconds.

Evading a NIDS

Attacker’s Goal: Evade NIDS Two attack vectors in an evasion attempt:

1st—alg. complexity attack targeting the NIDS

2nd—true attack targeting the network

Effect of an algorithmic complexity attack:(NIDS) Packets enter network unexamined(fail-closed IPS) Packets are dropped

Main results

In Snort, vulnerability in rule-matching worst-case vs. typical case: 6 orders of magnitude. “Backtracking Attack” Easily exploitable through packet payloads

Improved rule-matching algorithm limits running time differences to within 1 order of magnitude.

Outline

Snort rule matching Inducing backtracking attacks Countermeasures Measurement results Conclusion

Snort Rule Matching

content:”fmt=”; //P1

content:”player=”; //P3

content:”overflow”,relative; //P5

alert tcp $EXT_NET any -> $HOME_NET 99 (msg:”AudioPlayer jukebox exploit”;

pcre:”/^(mp3|ogg)/”,relative; //P2

pcre:”/.exe|.com/”,relative; //P4

sid:5678)

Snort Rule Matching

alert tcp $EXT_NET any -> $HOME_NET 99 (msg:”AudioPlayer jukebox exploit”; content:”fmt=”; //P1 pcre:”/^(mp3|ogg)/”,relative; //P2 content:”player=”; //P3 pcre:”/.exe|.com/”,relative; //P4 content:”overflow”,relative; //P5 sid:5678)

fmt=acc player=default fmt=mp3 rate=14kbps player=cmd.exe?overflow#@!%

Rule matches!

Matching the packetP1

P5

P4

P3

P2P2

Rule matches!

alert tcp $EXT_NET any -> $HOME_NET 99 (msg:”AudioPlayer jukebox exploit”; content:”fmt=”; //P1 pcre:”/^(mp3|ogg)/”,relative; //P2 content:”player=”; //P3 pcre:”/.exe|.com/”,relative; //P4 content:”overflow”,relative; //P5 sid:5678)

fmt=acc player=default fmt=mp3 rate=14kbps player=cmd.exe?overflow#@!%

Inducing Backtracking attacks

P1,P2,P3,P4 match in 3 positions each

P5 never matches

fmt=acc player=default fmt=mp3 rate=14kbps player=cmd.exe?overflow#@!%

fmt=mp3fmt=mp3fmt=mp3player=player=player=.exe.exe.exe

alert tcp $EXT_NET any -> $HOME_NET 99 (msg:”ReelAudio jukebox exploit”; content:”fmt=”; //P1 pcre:”/^(mp3|ogg)/”,relative; //P2 content:”player=”; //P3 pcre:”/.exe|.com/”,relative; //P4 content:”overflow”,relative; //P5 sid:5678)

Leads to excessive packet traversals!

Matching the malicious packet

alert tcp $EXT_NET any -> $HOME_NET 99 (msg:”AudioPlayer jukebox exploit”; content:”fmt=”; //P1 pcre:”/^(mp3|ogg)/”,relative; //P2 content:”player=”; //P3 pcre:”/.exe|.com/”,relative; //P4 content:”overflow”,relative; //P5 sid:5678)

fmt=mp3fmt=mp3fmt=mp3player=player=player=.exe.exe.exe

P5 P5 P5

P2

P4

P5 P5 P5

P4

P5 P5 P5

P4

P3

P5 P5 P5

P4

P5 P5 P5

P4

P5 P5 P5

P4

P2

P3

P1

P5 P5 P5

P4

P5 P5 P5

P4

P5 P5 P5

P4

P2

P3

P1

Are real rules vulnerable?

Rule numberProcessing

(s/GB)

Slowdown

Same proto All traffic

3682 (SMTP) 30,933,874 232,936X 1,501,644X

2611 (Oracle) 6,220,768 56,296X 301,979X

1382 (IRC) 1,956,858 134,031X 94,993X

2403 (NetBIOS) 357,777 490X 17,368X

1755 (IMAP) 89,181 444X 4,329X

Safer backtracking

Memoization: maintain a table of subproblem “answers”; never evaluate a predicate twice at the same starting payload offset

alert tcp $EXT_NET any -> $HOME_NET 99 (msg:”AudioPlayer jukebox exploit”; content:”fmt=”; //P1 pcre:”/^(mp3|ogg)/”,relative; //P2 content:”player=”; //P3 pcre:”/.exe|.com/”,relative; //P4 content:”overflow”,relative; //P5 sid:5678)

Identify constrained predicate sequences Monotone memoization: don’t re-evaluate monotone

predicates that have been evaluated at lower offsets

Reductions in processing cost

fmt=mp3fmt=mp3fmt=mp3player=player=player=.exe.exe.exe

P5 P5 P5

P2

P4

P5 P5 P5

P4

P5 P5 P5

P4

P3

P5 P5 P5

P4

P5 P5 P5

P4

P5 P5 P5

P4

P2

P3

P1

P5 P5 P5

P4

P5 P5 P5

P4

P5 P5 P5

P4

P2

P3

411

18

7 14 21

4650

54 4650

54 4650

54

28 35 42 28 35 42 28 35 42

Outline

Snort rule matching Inducing backtracking attacks Protecting against backtracking attacks Measurement results Conclusion

Measurement results

Rule number

Slowdown factor w.r.t. same protocol

Before w/ Memo+

3682 (SMTP) 232,936X 0.95X

2611 (Oracle) 56,296X 1.57X

1382 (IRC) 134,031X 6.00X

2403 (NetBIOS) 490X 0.17X

1755 (IMAP) 444X 0.46X

Live experiment topology

Background Traffic AC Attack True Attack

Live experiment

Background Traffic @ 10Mbps AC Attack

Targets Snort SMTP rule 3682Directed at sendmail server

True Attack: NIMDA300 exploit attempts, sent 1 byte per second.New exploit started every second.

Live experiment results

Attack DescriptionExploits

Detected

Required Rate (kbps)

Control (No attack) 300/300 --

2 packets every 60 s. 220/300 0.4

1 packet every 5 s. 4/300 2.4

1 packet every 3 s. 0/300 4.0

20 packets initially 0/300 0.8

1 packet every 3 s. 300/300 --

20 packets initially 300/300 --

Conclusions

NIDS operation is complex. Many opportunities for vulnerable algorithms.

In Snort, rule-matching is vulnerable and can be exploited by an attacker.

Memoization, along with other semantics-preserving operations, significantly reduces vulnerability.

Other vulnerable algoritms exist.

Backtracking Algorithmic Complexity Attacks Against a NIDS

Thank you.