detection of ascii malware parbati kumar manna dr. sanjay ranka dr. shigang chen
TRANSCRIPT
![Page 1: Detection of ASCII Malware Parbati Kumar Manna Dr. Sanjay Ranka Dr. Shigang Chen](https://reader030.vdocuments.site/reader030/viewer/2022033105/56649e925503460f94b977b8/html5/thumbnails/1.jpg)
Detection of ASCII Malware
Parbati Kumar Manna
Dr. Sanjay Ranka
Dr. Shigang Chen
![Page 2: Detection of ASCII Malware Parbati Kumar Manna Dr. Sanjay Ranka Dr. Shigang Chen](https://reader030.vdocuments.site/reader030/viewer/2022033105/56649e925503460f94b977b8/html5/thumbnails/2.jpg)
2
Internet Worm and Malware
• Huge damage potential Infects hundreds of thousands of
computers Costs millions of dollars in damage Melissa, ILOVEYOU, Code Red,
Nimda, Slammer, SoBig, MyDoom
• Mostly uses Buffer Overflow
• Propagation is automatic (mostly)
![Page 3: Detection of ASCII Malware Parbati Kumar Manna Dr. Sanjay Ranka Dr. Shigang Chen](https://reader030.vdocuments.site/reader030/viewer/2022033105/56649e925503460f94b977b8/html5/thumbnails/3.jpg)
3
Recent Trends
• Shift in hacker’s mindset
• Malware becoming increasingly evasive and obfuscative
• Emergence of Zero-day worms
• Arrival of Script Kiddies
![Page 4: Detection of ASCII Malware Parbati Kumar Manna Dr. Sanjay Ranka Dr. Shigang Chen](https://reader030.vdocuments.site/reader030/viewer/2022033105/56649e925503460f94b977b8/html5/thumbnails/4.jpg)
4
Motivation for ASCII Attacks
• Prevalence of servers expecting text-only input
• Text-based protocols
• Presumption of text being benign
• Deployment of ASCII filter for bypassing text
![Page 5: Detection of ASCII Malware Parbati Kumar Manna Dr. Sanjay Ranka Dr. Shigang Chen](https://reader030.vdocuments.site/reader030/viewer/2022033105/56649e925503460f94b977b8/html5/thumbnails/5.jpg)
5
IDS Detecting ASCII Attack?
• Disassembly-based IDS
All jump instructions are ASCII
Higher proportion of branches
Exponential disassembly cost
High processing overhead for IDS
• Frequency-based IDS
PAYL evaded by ASCII worm
![Page 6: Detection of ASCII Malware Parbati Kumar Manna Dr. Sanjay Ranka Dr. Shigang Chen](https://reader030.vdocuments.site/reader030/viewer/2022033105/56649e925503460f94b977b8/html5/thumbnails/6.jpg)
6
Buffer Overflow
![Page 7: Detection of ASCII Malware Parbati Kumar Manna Dr. Sanjay Ranka Dr. Shigang Chen](https://reader030.vdocuments.site/reader030/viewer/2022033105/56649e925503460f94b977b8/html5/thumbnails/7.jpg)
7
• Opcode Unavailability Shellcode requires binary opcodes Here only xor, and, sub, cmp etc. Must generate opcodes dynamically
• Difficulty in Encryption No backward jump Can’t use same decrypter routine
for each encrypted block No one-to-one correspondence
between ASCII and binary
Constraints of ASCII Malware
0 m a y v a r y
ASCII binary
![Page 8: Detection of ASCII Malware Parbati Kumar Manna Dr. Sanjay Ranka Dr. Shigang Chen](https://reader030.vdocuments.site/reader030/viewer/2022033105/56649e925503460f94b977b8/html5/thumbnails/8.jpg)
8
Creation of ASCII Malware
![Page 9: Detection of ASCII Malware Parbati Kumar Manna Dr. Sanjay Ranka Dr. Shigang Chen](https://reader030.vdocuments.site/reader030/viewer/2022033105/56649e925503460f94b977b8/html5/thumbnails/9.jpg)
9
Buffer Overflow using ASCII
Overflowing a buffer using an ASCII string:
![Page 10: Detection of ASCII Malware Parbati Kumar Manna Dr. Sanjay Ranka Dr. Shigang Chen](https://reader030.vdocuments.site/reader030/viewer/2022033105/56649e925503460f94b977b8/html5/thumbnails/10.jpg)
10
• Opcode Unavailability Dynamic generation of opcodes
needs more ASCII instructions for each binary instruction
• Difficulty in Encryption No backward jump means
decrypter block for each encrypted block must be hardcoded
Long sequence of contiguous valid instructions likely high MEL
Detection of ASCII Malware
What is this MEL?
![Page 11: Detection of ASCII Malware Parbati Kumar Manna Dr. Sanjay Ranka Dr. Shigang Chen](https://reader030.vdocuments.site/reader030/viewer/2022033105/56649e925503460f94b977b8/html5/thumbnails/11.jpg)
11
• Indicates maximum length of an execution path
Need to disassemble (and execute) from all possible entry points
All branching must be considered• Abstract payload execution
Used for binary worms with sled Effectiveness dwindled presently
Maximum Executable Length
![Page 12: Detection of ASCII Malware Parbati Kumar Manna Dr. Sanjay Ranka Dr. Shigang Chen](https://reader030.vdocuments.site/reader030/viewer/2022033105/56649e925503460f94b977b8/html5/thumbnails/12.jpg)
12
Benign Text has Low MEL
• Contains characters that correspond to invalid instructions
Privileged Instruction (I/O) Arbitrary Segment Selector More Memory-accessing
instructions – may use uninitialized registers
Long sequence of contiguous valid instructions unlikely low MEL
![Page 13: Detection of ASCII Malware Parbati Kumar Manna Dr. Sanjay Ranka Dr. Shigang Chen](https://reader030.vdocuments.site/reader030/viewer/2022033105/56649e925503460f94b977b8/html5/thumbnails/13.jpg)
13
Proposed Solution
Question:
• How long is “long”?
• Find out the maximum length of valid instruction sequence
• If it is long enough, the stream contains a malware
![Page 14: Detection of ASCII Malware Parbati Kumar Manna Dr. Sanjay Ranka Dr. Shigang Chen](https://reader030.vdocuments.site/reader030/viewer/2022033105/56649e925503460f94b977b8/html5/thumbnails/14.jpg)
14
• Toss a coin n times
• What is the probability that the max distance between two consecutive heads is ?
Probabilistic Analysis
Head (H) Invalid Instruction (I)
Tail (T) Valid Instruction (v)
T H T T H T T T T T H T T TV I V V I V V V V V I V V V
![Page 15: Detection of ASCII Malware Parbati Kumar Manna Dr. Sanjay Ranka Dr. Shigang Chen](https://reader030.vdocuments.site/reader030/viewer/2022033105/56649e925503460f94b977b8/html5/thumbnails/15.jpg)
15
Probabilistic Analysis
n = number of coin tosses p = probability of a head Xi = R.V.s for inter-head
distancesXmax = Max inter-head distanceC.D.F of Xmax = Prob [Xmax ≤ x]
= [1 – p(1-p)x ]n
F.P. rate = 1 - Prob [Xmax ≤ τ] = 1 - [1 – p(1-p)τ ]n
![Page 16: Detection of ASCII Malware Parbati Kumar Manna Dr. Sanjay Ranka Dr. Shigang Chen](https://reader030.vdocuments.site/reader030/viewer/2022033105/56649e925503460f94b977b8/html5/thumbnails/16.jpg)
16
Probabilistic Analysis
For a fixed N = k (exactly k invalid instructions)
![Page 17: Detection of ASCII Malware Parbati Kumar Manna Dr. Sanjay Ranka Dr. Shigang Chen](https://reader030.vdocuments.site/reader030/viewer/2022033105/56649e925503460f94b977b8/html5/thumbnails/17.jpg)
17
Probabilistic Analysis
For all possible values of N:
![Page 18: Detection of ASCII Malware Parbati Kumar Manna Dr. Sanjay Ranka Dr. Shigang Chen](https://reader030.vdocuments.site/reader030/viewer/2022033105/56649e925503460f94b977b8/html5/thumbnails/18.jpg)
18
Threshold Calculation
n , p , (false positive rate)
(max inter-head distance)
Known
Unknown
)1log(
log))1(1log(1
p
pn
Threshold
![Page 19: Detection of ASCII Malware Parbati Kumar Manna Dr. Sanjay Ranka Dr. Shigang Chen](https://reader030.vdocuments.site/reader030/viewer/2022033105/56649e925503460f94b977b8/html5/thumbnails/19.jpg)
19
Independence Assumption
2 test contingency table
Observed Expected
I2 is valid
I2 is invali
d
I1 is valid
I2 is invalid
I1 is valid 8960 2797 8922 2835
I1 is invalid 2797 938 2835 900
• Validity of an instruction is an independent event
• All the Xi’s are independent (while Xi = n)
![Page 20: Detection of ASCII Malware Parbati Kumar Manna Dr. Sanjay Ranka Dr. Shigang Chen](https://reader030.vdocuments.site/reader030/viewer/2022033105/56649e925503460f94b977b8/html5/thumbnails/20.jpg)
20
Threshold Calculation
With increasing n, we must choose a larger to keep the same rate of false positive
![Page 21: Detection of ASCII Malware Parbati Kumar Manna Dr. Sanjay Ranka Dr. Shigang Chen](https://reader030.vdocuments.site/reader030/viewer/2022033105/56649e925503460f94b977b8/html5/thumbnails/21.jpg)
21
Threshold Calculation
With decreasing p, we must choose a larger to keep the same rate of false positive
![Page 22: Detection of ASCII Malware Parbati Kumar Manna Dr. Sanjay Ranka Dr. Shigang Chen](https://reader030.vdocuments.site/reader030/viewer/2022033105/56649e925503460f94b977b8/html5/thumbnails/22.jpg)
22
Determine n
size)n instructio (average
)charactersinput ofnumber (
I
Cn
E[I] = E[Prefix chain length] + E[core instruction length]
Obtained from character frequency of input data
![Page 23: Detection of ASCII Malware Parbati Kumar Manna Dr. Sanjay Ranka Dr. Shigang Chen](https://reader030.vdocuments.site/reader030/viewer/2022033105/56649e925503460f94b977b8/html5/thumbnails/23.jpg)
23
1.Privileged instructions
2.Wrong Segment Prefix Selector
3.Un-initialized memory access
Determine p
Invalid Instructions
Only 1. and 2. can be determined on a standalone basis
![Page 24: Detection of ASCII Malware Parbati Kumar Manna Dr. Sanjay Ranka Dr. Shigang Chen](https://reader030.vdocuments.site/reader030/viewer/2022033105/56649e925503460f94b977b8/html5/thumbnails/24.jpg)
24
Experimental Setup
![Page 25: Detection of ASCII Malware Parbati Kumar Manna Dr. Sanjay Ranka Dr. Shigang Chen](https://reader030.vdocuments.site/reader030/viewer/2022033105/56649e925503460f94b977b8/html5/thumbnails/25.jpg)
25
Implementation
![Page 26: Detection of ASCII Malware Parbati Kumar Manna Dr. Sanjay Ranka Dr. Shigang Chen](https://reader030.vdocuments.site/reader030/viewer/2022033105/56649e925503460f94b977b8/html5/thumbnails/26.jpg)
26
Experimental Setup
• Benign data setup ASCII stream captured from live CISE network
using Ethereal
• Malicious data setup Existing framework used to generate ASCII worm
by converting binary worms
• Promising experimental results for max valid instruction length Benign: all max values all below threshold Malicious: values significantly higher than
![Page 27: Detection of ASCII Malware Parbati Kumar Manna Dr. Sanjay Ranka Dr. Shigang Chen](https://reader030.vdocuments.site/reader030/viewer/2022033105/56649e925503460f94b977b8/html5/thumbnails/27.jpg)
27
Experimental Results (DAWN)
![Page 28: Detection of ASCII Malware Parbati Kumar Manna Dr. Sanjay Ranka Dr. Shigang Chen](https://reader030.vdocuments.site/reader030/viewer/2022033105/56649e925503460f94b977b8/html5/thumbnails/28.jpg)
28
Experimental Results (APE-L)
![Page 29: Detection of ASCII Malware Parbati Kumar Manna Dr. Sanjay Ranka Dr. Shigang Chen](https://reader030.vdocuments.site/reader030/viewer/2022033105/56649e925503460f94b977b8/html5/thumbnails/29.jpg)
29
Contrasting with APE
• Full content examination
• Threshold calculation
• Sled Vs. malware
• Exploiting text-specific properties
![Page 30: Detection of ASCII Malware Parbati Kumar Manna Dr. Sanjay Ranka Dr. Shigang Chen](https://reader030.vdocuments.site/reader030/viewer/2022033105/56649e925503460f94b977b8/html5/thumbnails/30.jpg)
30
Multilevel Encryption
Encryption
Decryption
binary ASCII ASCII
ASCII ASCII binary
Only Visible decrypter
![Page 31: Detection of ASCII Malware Parbati Kumar Manna Dr. Sanjay Ranka Dr. Shigang Chen](https://reader030.vdocuments.site/reader030/viewer/2022033105/56649e925503460f94b977b8/html5/thumbnails/31.jpg)
31
Multilevel Encryption
Text0x20 – 0x3F
Text0x40 – 0x5F
Text0x60 – 0x7E
Binary
Binary
![Page 32: Detection of ASCII Malware Parbati Kumar Manna Dr. Sanjay Ranka Dr. Shigang Chen](https://reader030.vdocuments.site/reader030/viewer/2022033105/56649e925503460f94b977b8/html5/thumbnails/32.jpg)
32
Questions
![Page 33: Detection of ASCII Malware Parbati Kumar Manna Dr. Sanjay Ranka Dr. Shigang Chen](https://reader030.vdocuments.site/reader030/viewer/2022033105/56649e925503460f94b977b8/html5/thumbnails/33.jpg)
33
Thank you