Malware Analysis

Malware is software designed to infiltrate or damage systems. Understanding how attackers classify and hide malware helps defenders build better detection tools.

• Viruses – Infect legitimate programs; replicate when the host runs
• Worms – Spread over the network without user action
• Trojans – Disguised as legitimate software; users install them willingly
• Botnets – Networks of compromised machines controlled remotely
• Spyware / adware – Harvest data or push unwanted content

Signature-based: byte sequences in malware. Manual analysis, reverse engineering. Syntactic signatures easily evaded.

Malware authors use various techniques to evade detection. These fall into three broad categories depending on what the defender is doing.

• Against signature detection: Polymorphism, metamorphism—change the binary so byte-pattern signatures no longer match
• Against dynamic analysis: Anti-debugging, anti-VM, emulator detection—detect sandbox environments and refuse to run
• Against static analysis: Anti-disassembly, packing, control-flow obfuscation—make code hard to inspect without executing it

Polymorphic malware encrypts its main body and uses a different encryption key for each copy. The decryptor (which must run in the clear) may have several variants or be obfuscated. Byte-sequence signatures on the body fail because the ciphertext changes every time.

• Encrypted body – The real payload is encrypted; each instance uses a different key
• Varying decryptor – A few decryptor variants, or obfuscation, make it harder to signature
• Why signatures fail – The encrypted bytes change; the decryptor may be small and variable

Metamorphic malware avoids encryption entirely. Instead, it rewrites its own code so each instance looks different but behaves the same. The entire body is obfuscated through transformations like code reordering, garbage insertion, equivalent instruction replacement, jump insertion, and packing.

• Polymorphic vs metamorphic: Polymorphic encrypts the body; metamorphic rewrites it in-place
• Transformations: Register renaming, no-op insertion, instruction reordering, equivalent opcode substitution
• Example: W32/Simile used register renaming, no-op insertion, and instruction reordering to generate thousands of unique variants

These techniques make static inspection (disassembly, decompilation) difficult or misleading.

• Anti-disassembly: Mix code and data so the disassembler misinterprets bytes; use indirect jumps so control flow is unclear; exploit variable-length x86 instructions so alignment is ambiguous
• Self-modifying code: Generate or decrypt new code at runtime—the real code is invisible until execution
• Packing: Encapsulate the real payload in a compressed or encrypted form; unpack only when the program runs

Running malware in a sandbox (e.g., VM) defeats most anti-static techniques—the real code must execute to be observed. However, malware can detect analysis environments and refuse to run.

• VM detection: Nopill, Vmdetect, Redpill—check CPU flags, timing, or artifacts that betray a hypervisor
• Debugger detection: IsDebuggerPresent, timing checks—detect if a debugger is attached and exit or behave differently
• System-call tracing: DLL hooking, kernel drivers, or VMM-level interception (e.g., CWSandbox, TTAnalyze) to log behavior

Packed malware hides its real code until runtime. Unpackers automatically reveal the hidden code so it can be analyzed statically or used for signatures.

• PolyUnpack: Builds a static model of the original executable; when execution diverges (new code runs that wasn't in the model), that code is considered unpacked—single-step and follow the instruction pointer (EIP)
• Renovo / OmniUnpack: Detect when code that was written to memory is then executed—that is the unpacked payload; fine-grained (per instruction) or coarse (per page or syscall)
• Common packers: UPX, Armadillo, commercial protectors—often add anti-debug, anti-trace, and obfuscation

Malware often hides malicious behavior behind a trigger—a condition that may not be met during a short sandbox run. A single execution path misses the hidden code.

• Timebombs: Act only after a date or delay; sandbox may not run long enough
• Logic bombs: Variable-based conditions (e.g., "run only if file X exists")
• Bot commands: Malicious behavior triggered by remote commands; no command = benign behavior
• Conditional obfuscation (Sharif et al.): The key or condition is not in the code; e.g., if (Hash(cmd)==H)—finding the right cmd requires solving a hash, which is hard for symbolic execution