The battle between defenders and attackers is a constant arms race. As we fight malware on our desktops, the landscape is changing and growing into something entirely different. We see our newest improvements in technologies getting applied in ways that also improve threats. One of these technologies is standing out from the others as it matures. Self-virtualizing malware is on the horizon and we need to be prepared for it.
Generally, we have only seen virtualization occurring on a system-wide scale where an entire operating system is encapsulated within another. This scoping capability enables a perfect opportunity to watch a system in its entirety. Malware programs can take this concept and apply it to a smaller scale. If we were to look at a self-virtualizing malware binary, what would it look like?
A self-virtualizing binary would have the ability to separate its intentions from what an analysis would be looking for. The virtualization would be an abstraction layer between real machine language and a new pseudo machine. This abstraction layer would not defeat a good analyst by itself, but if combined with anti-debugging features it could easily defeat many. So the analyst would have to detect the bootstrapping code of the malware.
After identifying the bootstrap code, the analyst would likely have to circumvent another round of anti-debugging tricks in the pseudo language. This is less likely to be a problem than standard anti-debugging tricks since the abstraction language is customized and the debugger will not be making assumptions about the code. If the malware were to abstract itself many times, then the debugger would want to develop a script to decode the abstraction repeatedly. This script would be the area where the second set of anti-debugging features would exist.
Even though this malware can abstract itself into its own virtual machine, it is still victim to the analyst acting outside of the binary. Even if the binary is not immediately disassembled, it can generally be observed and monitored. This behavior may give some clues to portions of its code. Any interaction of the malware with other processes or systems would have to leave the abstraction later to communicate or act. This is where the true observation would occur. Consider a malware that sends a network packet. For the packet to get to the wire it must be handed off to the host operating system.
The key to defeating self-virtualizing malware is to have a foothold in a known-good environment. By positioning our analysts and defenses outside the scope of the malware we can observe the bootstrapping and the affects it. Regardless of how complicated the malware is, we can still intercept and defend when the malware tries to act out in the real world.