intrusion detection and malware analysis - introduction to ... · malware: definitions and root...

Intrusion Detection and Malware AnalysisIntroduction to Malware

Pavel LaskovWilhelm Schickard Institute for Computer Science

Malware: definitions and root causes

The term malicious software denotes program code executedwithout a user’s consent and carrying out harmful functionality.

Why is malware such a serious problem?Technical reasons:

Difficulty to separate code and dataHomogeneity of “computing base” (Windows, iPhone,Facebook)Unprecedented connectivity

Non-technical reasons:

Recklessness and ignorance of usersProfitability of security violations (spam, DoS extortion,stealing of personal information)

Malware: definitions and root causes

The term malicious software denotes program code executedwithout a user’s consent and carrying out harmful functionality.

Why is malware such a serious problem?

Technical reasons:Difficulty to separate code and dataHomogeneity of “computing base” (Windows, iPhone,Facebook)Unprecedented connectivity

Non-technical reasons:

Recklessness and ignorance of usersProfitability of security violations (spam, DoS extortion,stealing of personal information)

Malware: definitions and root causes

The term malicious software denotes program code executedwithout a user’s consent and carrying out harmful functionality.

Why is malware such a serious problem?Technical reasons:

Difficulty to separate code and dataHomogeneity of “computing base” (Windows, iPhone,Facebook)Unprecedented connectivity

Non-technical reasons:

Recklessness and ignorance of usersProfitability of security violations (spam, DoS extortion,stealing of personal information)

Malware: definitions and root causes

The term malicious software denotes program code executedwithout a user’s consent and carrying out harmful functionality.

Why is malware such a serious problem?Technical reasons:

Difficulty to separate code and dataHomogeneity of “computing base” (Windows, iPhone,Facebook)Unprecedented connectivity

Non-technical reasons:

Recklessness and ignorance of usersProfitability of security violations (spam, DoS extortion,stealing of personal information)

An example from last semester

An example from last semester

“Theoretical foundations” of malware

Von Neumann’s model (1948,1953)universal machineuniversal constructorinformation on the tape

Darwin/Core Wars: fighting programs (1966)a special assembly language with 10 instructions (“Redcode”)two programs simultaneously running in the same memorya program dies if it executes division by 0 or a null instructionto increase their survival chances, programs can replicatethemselves

Cohen’s thesis (1984)definition of a virusmathematical description of virus propagation

Pioneers of practical malware

Elk Cloner (1982)an Apple-II program written by a student Rich Skrentaa program spread via boot sector infectionon every 50-th reset a short poem was displayed by hookingthe reset handler

Brain Virus (1986)first virus to spread in the wildwritten by Ashar brothers to prevent illegal copying of softwareincluded in a boot sector of distrubuted software diskettes

Morris worm (1988)used a debugging feature of sendmail (remote execution)propagated in the Arpanetpenetrated ca. 6,000 computers (10% of Arpanet)

First generation of malware

DefinitionA virus is a program that infects other programs by modifyingthem to include a possibly modified copy of itself.

Boot sector virusesstored in disk boot sectors, executed automatically at boot

Executable virusesstored in exetuable data, hijack flow control

Macro virusesstored in non-executable data, started by opening a document

Second generation of malware

DefinitionA worm is a program that actively propagates over computernetworks, with or without human interaction.

Mass-mailers (e.g. Melissa, LoveLetter, Nimda)use SMTP protocol for propagationusually (but not necessarily) require manual interaction

In-memory worms (e.g. Slammer, CodeRed)proliferation over TCP/HTTPautomatic execution on vulnerable systems (very fast!)

Third generation of malware (current)

DefinitionA trojan horse is a program carrying a hidden functionality behinda seemingly useful one.

focus on stealthy operationanti-deteciton mechanisms (obfuscation, encryption andpolymorphism)combination of various malware types and techniquesincreasing commercialization

Further malware types (1)

Backdoor: remote access to compromized computers(especially used in botnets)Downloader: installation of malicious content (used toovercome filtering of executables and update of alreadyinstalled malware)Rootkit: modification of system behavior, either at a userlevel (e.g. user settings) or a root level (e.g. system tools orregistry entries)Dialer: calling premium services, re-surging in mobilemalware

Further malware types (2)

Spyware: monitoring of user behavior (e.g. during webbrowsing)Adware: unsolicited presentation of advertisementKeylogger: capturing of keystrokes, especially for typed inpasswords or financial credentialsSniffer: capturing of network traffic, especially for passwordssend in clear text

Malware carriers: boot sector

Save the original MBR in a safe locationOverwrite the MBR with an infected oneBootstrap a system using the new MBR

Viren-Infektionsvektoren Infektion des Boot-Sektors

Infektion des Boot-Sektors

Virus kopiert originalen Bootblock und überschreibt den BootblockVirus erhält Kontrolle beim Bootvorgang noch bevorBetriebssystem und Anti-Viren-SW geladen werdenGegenmaßnahme: Bootblockschutz im BIOS

Abbildung: Mehrfache Infektion durch verschiedene Viren

c© Ulrich Flegel Reaktive Sicherheit · Teil 4 21 / 130

Malware carriers: COM executables

Append a virus body to a programSave an entry point to a program in a virus bodyReplace a program entry point with a jump to a virus bodyVirus code restores the original entry point and jumps to itafter its own execution

Viren-Infektionsvektoren Infektion von Dateien

Anhängen an die Datei (1)

Abbildung: Infektion durch Anhängen

c© Ulrich Flegel Reaktive Sicherheit · Teil 4 30 / 130

Malware carriers: EXE executables

Append a virus body to a programOverwrite a program header to switch the entry point to avirusJump to the original entry point during execution

Viren-Infektionsvektoren Infektion von Dateien

Anhängen an die Datei (2)

Abbildung: Infektion durch Anhängen

c© Ulrich Flegel Reaktive Sicherheit · Teil 4 31 / 130

Malware carriers: PE executables

Virus is split into parts that areparcelled together by adefragmentation routine.Defragmentation routine isinserted between the PE headerand the first section.Entry point in the PE header isswitched to the defragmentationroutine.Additional viral sections are storedin unused parts of sections.

Viren-Infektionsvektoren Infektion von Dateien

Einstreuen in ungenutzte Füllbereiche (1)Code Interlacing Infection bzw. Cavity Hole Infection

Abbildung: Infektion durch Einstreuen in ungenutzte Dateibereiche

c© Ulrich Flegel Reaktive Sicherheit · Teil 4 34 / 130

Malware carriers: companion viruses

MS-DOS feature: if a command is typed without anextension, first a .COM file is searched for and then an .EXE

Exploitation: Inject a malicions program <program>.COMwhich calls <program.EXE> besides doing its dirty job.

Variations on a theme:Renaming legitimate filesPlacing a new earlier in the search path

Malware carriers: companion viruses

MS-DOS feature: if a command is typed without anextension, first a .COM file is searched for and then an .EXEExploitation: Inject a malicions program <program>.COMwhich calls <program.EXE> besides doing its dirty job.

Variations on a theme:Renaming legitimate filesPlacing a new earlier in the search path

Malware carriers: companion viruses

MS-DOS feature: if a command is typed without anextension, first a .COM file is searched for and then an .EXEExploitation: Inject a malicions program <program>.COMwhich calls <program.EXE> besides doing its dirty job.

Variations on a theme:Renaming legitimate filesPlacing a new earlier in the search path

Malware carriers: macros

Malicious functionality is implemented in Visual Basic forApplications (VBA).If a document template are infected, so will be everydocument on a system.

Viren-Infektionsvektoren Infektion von Dateien

Dokumenten-Viren (3)Macro Viruses

Abbildung: Beispiel: Concept Virus infiziert Microsoft-Word-Dateien

c© Ulrich Flegel Reaktive Sicherheit · Teil 4 42 / 130

Malware carriers: Visual Basic (VB) script

Similar functionality to macro viruses implemented in “pure”VB.A script is executed by a user by clicking at the emailattachment (LoveLetter mass-mailer).Why does the user execute a VB script? The attachmentname LOVE-LETTER-FOR-YOU.TXT.vbs is shown inOutlook as LOVE-LETTER-FOR-YOU.TXT (withoutextension!).Further functionality is possible via access to ActiveXobjects.

Malware carriers: SuperLogo

A set of “turtle-control” commandsis provided in the Logo language(e.g. HIDETURTLE, FORWARD,PENUP, PENDOWN, WAIT).Sets of commands can be saved inLGP file.A command PRINTTO “XYZ” canbe used to write directly to files(with a complete path to a file).

Overwrite winstart.bat with anyfunctionality and it will be run!

Malware carriers: SuperLogo

A set of “turtle-control” commandsis provided in the Logo language(e.g. HIDETURTLE, FORWARD,PENUP, PENDOWN, WAIT).Sets of commands can be saved inLGP file.A command PRINTTO “XYZ” canbe used to write directly to files(with a complete path to a file).Overwrite winstart.bat with anyfunctionality and it will be run!

Malware carriers: JavaScript

Similar functionality to VB scripts:Read, write and overwrite filesAccess to ActiveX communication objects: send emails...

Automatic invocation during web browsing: infection ispossible via opening of malicious web pages.

Malware carriers: other script languages

Potential carriers: shell scripts, PHP, Perl, Python, Emacs,Tcl, VIMInfection strategies:

file overwriting“exec” functionality

Covering the tracks

Erasing dataManipulation of registry entriesChanging time stampsChanging checksumsMemory-resident operationAPI hooking...

Malware packers

A run-time executable packer is a program for compressing aprogram code and data.Primary goal: reduction of the program sizeSide effect: signature detection is heavily impeded!Main idea:

Compress the original executable program and store it in adata segmentPut a decompression routine in some program segment.Decompression routine unpacks malicious code into memoryand transfers control to it.

Further development: compression can be implementedusing cryptographic primitives of varying strength.

Advanced evasion techniques

Oligomorphism: a set of decryptors is carried along; a newdecryptor is chosen at random for each new replication.Polymorphism: insert junk or semantically equivalent codeinto a decryptor to increase its variability.Metamorphism: instead of decryption, the malware body ismutated during each replication using equivalent programtransformations.

Lessons learned

The key challendge in fighting malware is its rapid evolutiondue to an underlying monetary interest.The main technical cause of malware is poor separationbetween code and data.Malware exemplifies a fundamental tradeoff betweensecurity and usability: advanced programming languagefeatures facilitate operation of malware.

Recommended reading

John Aycock.Computer Viruses and Malware.Springer, 2006.

Peter Szor.The art of computer virus research and defense.Symantec Press, 2005.