Static Devirtualization of Tencent VM: Breaking ACE Anti-Cheat Obfuscation with Guided Symbolic Execution

Executive Summary

Virtual machine obfuscation has been the default answer for a decade whenever a vendor needed to make a binary genuinely painful to read. The premise is simple: rather than shipping the real AMD64 instructions of a sensitive routine, you translate them into a private bytecode and ship an interpreter alongside it. An analyst staring at the protected function no longer sees the algorithm — they see a dispatch loop and an opaque byte stream. Tencent built exactly such a VM and deployed it inside the ACE anti-cheat kernel drivers, hardening it further with two features most academic VM obfuscators never bother with: correct behaviour on Intel CET hardware, and full compatibility with Windows structured exception handling and stack unwinding.

The Aftermath Labs write-up argues that none of that engineering effort actually raises the cost of a serious attack. Once an analyst lifts the virtualized code into an SSA intermediate representation and lets standard compiler optimizations run — constant promotion, constant folding, and a handful of trivial mixed boolean-arithmetic reduction rules — the VM dissolves. The bytecode is folded into constants, the decryption chain collapses, the virtualized conditional branches pattern-match back into native JCCs, and what is left is the original function. Applied across four ACE kernel drivers, the technique recovered 815 out of 865 virtualized functions, a 94.2% coverage rate, with the output verified independently by a third party. This article walks through the VM’s architecture, its CET and SEH mechanisms, the guided symbolic execution pipeline, the pattern rules that undo virtualized control flow, the lowering step that rebuilds a correct stack frame, and the failure cases that remain.

Prerequisite Material

The original article lists three references as background reading:

• https://k0mkc.hatenablog.com/entry/2026/07/29/042406
• https://arxiv.org/abs/2603.18355
• https://arxiv.org/abs/1909.01752

Legal Disclaimer

This work constitutes reverse engineering, decompilation, and de-virtualization performed solely for the purpose of achieving interoperability with ACE-protected software on Linux and Proton environments. Such activities are undertaken in good faith under applicable exceptions to copyright law, including those recognizing the right to reverse engineer software for interoperability purposes.

Any sharing has been limited exclusively to trusted third parties for independent technical validation. The original protected software remains subject to copyright and the protections of the Digital Millennium Copyright Act (DMCA) and other applicable laws.

Aftermath Labs, “Static Devirtualization of Tencent VM”

Introduction

Interest in Tencent’s VM obfuscation has climbed noticeably over the past several months. The Aftermath Labs team states they have had complete static devirtualization of this particular VM working for quite some time, and that other researchers have independently reached comparable deobfuscation results. Their broader framing is that the steady advance of AI-assisted analysis is exposing a gap that was always there but easy to ignore: the distance between obfuscation that is genuinely strong and obfuscation that merely looks intimidating.

Their long-standing position is that the classic style of virtual machine obfuscator does not hold up against an attacker who has a flexible lifting and recompilation framework. A great many companies have attempted to build in-house obfuscation solutions; most of them are not particularly strong, and most of them are already being deobfuscated by private groups who simply do not publish. The purpose of the article is to explain the Tencent virtual machine in detail, document how it manages to support CET and SEH correctly, and demonstrate why it is weak in the face of guided symbolic evaluation.

A reader of their previous article, Static Devirtualization of Themida, had asked about coverage statistics and version-related questions. Version information for Tencent VM is not available to them, but detailed static devirtualization coverage numbers are provided here, and a later section covers that formally. The output files were handed to a third party — Daax from secret club — who can independently confirm the cleanliness and coverage of the recovered code.

Virtual Machine Architecture

The Tencent VM is structured so that the entry point of the original function jumps directly into the .tvm section. On arrival, stack space is allocated for the VM context, and every general-purpose register plus EFLAGS is saved into that context — first pushed onto the stack, then moved into the VM context proper. Anyone who has looked at VMProtect will find this immediately familiar: VMProtect likewise pushes all GPRs to the stack, and its first few VM handlers then pop them back off and store them into the VM context area.

From that point execution threads in and out of a shared virtual machine dispatch loop. The loop is the hub through which every handler is reached, and control leaves it only for two reasons: to execute calls, and to execute what the authors term boxed instructions. That second category turns out to be the single most useful structural weakness in the entire design.

Boxed Instructions

The Tencent VM models only a subset of AMD64. Anything falling outside that subset is handled by a boxed instruction: the VM performs a full context restoration and then executes the original instruction natively. At that precise moment the machine state is indistinguishable from what the unvirtualized function would have produced, so the instruction executes with entirely correct semantics without the VM ever needing to model it. Once the instruction retires, the VM captures register state back into the VM context and resumes dispatch.

More usefully for an attacker, every boxed instruction is a point at which the VM is forced to materialize real guest state. That makes it a reliable sink point — a place where the abstraction leaks by design and cannot avoid leaking. The devirtualization pipeline relies on exactly this property when recovering the original stack frame size, as described in the lowering section below.

It is worth noting that some mechanism for executing native instructions inside virtualized functions is a practical necessity, not a design shortcut. Consider an instruction such as CPUID: it would be impossible to model inside the VM without actually executing CPUID on the real processor. The same applies to other architectural instructions such as RDMSR and WRMSR. Any VM obfuscator targeting kernel-mode code will need boxed instructions, and every one of them is a window into the guest state.

CET Compatibility

Entering the dispatcher with a CALL creates a problem on hardware that implements Control-flow Enforcement Technology. Under CET a CALL pushes the return address onto a shadow stack in addition to the ordinary data stack, and a RET pops from both and compares them. The VM’s calls into the dispatcher never return. Left alone, the shadow stack would grow monotonically for the entire lifetime of the virtualized function and eventually fault.

Tencent VM handles this at runtime rather than at protection time, which is the more elegant of the two options. It executes RDSSPQ to read the current shadow stack pointer. RDSSPQ is encoded in the hint-NOP space, so on a processor — or in a process — without shadow stacks enabled it retires as a NOP and leaves its destination register untouched. Zeroing the register beforehand and testing it afterward is therefore a feature check that is correct on old and new hardware alike, and costs essentially nothing on either.

When the check indicates that CET is active, the VM issues INCSSPQ 2 (with the register containing 2), advancing the shadow stack pointer by two entries and discarding the two return addresses that no matching RET will ever consume.

SEH Compatibility

The exception handling story is considerably more involved, and it is the part of the design that shows the most care. The VM is covered by a single large .pdata entry whose unwind info spans the entire VM range — the VM entry, the dispatcher, and all the handlers. That unwind info carries a language-specific exception handler for the VM.

Alongside it sits a never-executed function carrying what the authors call phantom unwind info: a function whose sole purpose is to describe unwind operations, never to run. The VM entry reserves a 0x68 byte local area, of which a 0x48 byte block is used exclusively during unwinding and holds all nine non-volatile registers. The phantom unwind info attached to this dummy function describes the saves of all non-volatile registers, and the address of that dummy function is kept in [RBP+0] — one of the slots the VM entry reserves — for the entire lifetime of VM execution.

The VM entry’s own unwind info carries a UWOP_SET_FPREG with RBP as the frame register, so unwinding is performed relative to RBP. RBP is saved at VM entry and from that moment is never used as scratch during VM execution, which means the unwinder can correctly unwind from any point inside the VM — an invariant the obfuscator must hold for the whole dispatch loop and every handler.

When an exception is raised inside the VM, the VM’s exception handler runs. It copies every guest non-volatile register into the save area reserved at VM entry, and overwrites the unwind target slot located 8 bytes below the guest RSP with the guest RIP. The handler then returns ExceptionContinueSearch.

The unwinder then follows [RBP+0] and picks up the dummy function as the next entry. Its .pdata is looked up, and through that phantom unwind info every guest non-volatile register the handler has just staged is written back into the native register in the CONTEXT record where it ultimately belongs.

Finally the unwinder reads the contents of the unwind target slot as the RIP and advances RSP by 8, so that CONTEXT->RIP holds the guest RIP and CONTEXT->RSP lands exactly on the guest RSP. The original function’s virtual unwind then continues as a guest state, entirely unaware that a virtual machine was ever involved. Boxed instructions are given chained unwind info as needed, because they run outside the VM proper.

Guided Symbolic Execution

Guided symbolic execution is the process of lifting native instructions — AMD64 in this case — up to a higher level SSA intermediate representation that can be easily optimized and manipulated. The objective is to symbolically evaluate the entire virtualized function so that an IR function is produced containing all of the semantics of the original routine. To make that work, the symbolic lifting loop needs guidance whenever indirect control flow is discovered. Classical virtual machine obfuscation uses bytecode to influence indirect control flow inside the machine, encoding which VM handlers execute in what order. In Tencent VM, register R11 holds the address of the virtual machine bytecode that the interpreter consumes.

Guided symbolic execution achieves its results by feeding the engine obfuscation-specific information. In the case of Tencent VM the goal is to symbolically inline the call to the VM dispatcher loop. A simple but effective heuristic is to follow calls that have an int3 placed directly after them. For whatever reason, every single one of these calls along the symbolic evaluation path is a call into the VM dispatcher loop. The alternative is to declare the VM dispatcher function to the symbolic evaluation engine as a valid call target to follow and inline.

To solve indirect control flow inside the virtual machine, the bytecode must be promoted to a constant within the SSA IR so that the remaining optimizations can fold the bytecode decryption operations away. This promotion of load operations has to be carefully scoped: promote too aggressively and original semantic load operations get turned into constants too, which silently corrupts the recovered function.

Preventing re-lifting — effectively unrolling — of virtualized loops requires tracking the VIP, so that if the next VIP or VM handler has already been lifted, a backedge is created to it instead. VIP tracking is obfuscation specific. For Tencent VM the VIP is held inside the VM context structure, and the offset at which it lives can be dynamically resolved with an algorithm that finds the last stored value into the VM context whose value is a pointer into the .tvm0 section, since the bytecode address will fall inside that range. This heuristic works well and reveals the VIP automatically, without hardcoding an offset that would break on the next build.

Virtualized Conditional Control Flow

Conditional control flow inside the Tencent VM is implemented by expanding the flag comparison operations normally performed by native JCC instructions into multiple VM handlers. When symbolic evaluation halts on indirect control flow with a symbolic destination, it means one of two things: either the lifter is stopped at a virtualized JCC, or the optimizations are incomplete and something failed to fold.

Virtualized JCC logic can be transformed back into a native JCC using pre-defined IR SSA DAGs. If the current indirect control flow matches one of those pre-defined JCC DAGs, a rewrite is performed. A useful side effect is that during the same rewrite step the branch targets can be extracted directly from the IR, because the DAG implicitly defines where the branch destinations are.

The complete pattern set covers CF, PF, ZF, SF and OF, in both the zero and mask forms, for both the e and ne result kinds:

; ── CF ── mask 0x1, idx 0
pattern vjcc.cf.ae.zero { body(0x1,   0x0)   ; %r = R{e}  %z } => { %r = R{ae} %cf }
pattern vjcc.cf.b .zero { body(0x1,   0x0)   ; %r = R{ne} %z } => { %r = R{b}  %cf }
pattern vjcc.cf.b .mask { body(0x1,   0x1)   ; %r = R{e}  %z } => { %r = R{b}  %cf }
pattern vjcc.cf.ae.mask { body(0x1,   0x1)   ; %r = R{ne} %z } => { %r = R{ae} %cf }

; ── PF ── mask 0x4, idx 1
pattern vjcc.pf.np.zero { body(0x4,   0x0)   ; %r = R{e}  %z } => { %r = R{np} %pf }
pattern vjcc.pf.p .zero { body(0x4,   0x0)   ; %r = R{ne} %z } => { %r = R{p}  %pf }
pattern vjcc.pf.p .mask { body(0x4,   0x4)   ; %r = R{e}  %z } => { %r = R{p}  %pf }
pattern vjcc.pf.np.mask { body(0x4,   0x4)   ; %r = R{ne} %z } => { %r = R{np} %pf }

; ── ZF ── mask 0x40, idx 3
pattern vjcc.zf.ne.zero { body(0x40,  0x0)   ; %r = R{e}  %z } => { %r = R{ne} %zf }
pattern vjcc.zf.e .zero { body(0x40,  0x0)   ; %r = R{ne} %z } => { %r = R{e}  %zf }
pattern vjcc.zf.e .mask { body(0x40,  0x40)  ; %r = R{e}  %z } => { %r = R{e}  %zf }
pattern vjcc.zf.ne.mask { body(0x40,  0x40)  ; %r = R{ne} %z } => { %r = R{ne} %zf }

; ── SF ── mask 0x80, idx 4
pattern vjcc.sf.ns.zero { body(0x80,  0x0)   ; %r = R{e}  %z } => { %r = R{ns} %sf }
pattern vjcc.sf.s .zero { body(0x80,  0x0)   ; %r = R{ne} %z } => { %r = R{s}  %sf }
pattern vjcc.sf.s .mask { body(0x80,  0x80)  ; %r = R{e}  %z } => { %r = R{s}  %sf }
pattern vjcc.sf.ns.mask { body(0x80,  0x80)  ; %r = R{ne} %z } => { %r = R{ns} %sf }

; ── OF ── mask 0x800, idx 5
pattern vjcc.of.no.zero { body(0x800, 0x0)   ; %r = R{e}  %z } => { %r = R{no} %of }
pattern vjcc.of.o .zero { body(0x800, 0x0)   ; %r = R{ne} %z } => { %r = R{o}  %of }
pattern vjcc.of.o .mask { body(0x800, 0x800) ; %r = R{e}  %z } => { %r = R{o}  %of }
pattern vjcc.of.no.mask { body(0x800, 0x800) ; %r = R{ne} %z } => { %r = R{no} %of }

Those twenty patterns collapse neatly into a single parameterized template plus five instantiations, one per flag, which is the form worth carrying into a real rule engine:

template vjcc<FLAG, MASK, IDX, CC_SET, CC_CLEAR> {
  %rf = X86ReadFlags %f[0..5]
  %w  = launder %rf
  %m  = And %w, imm MASK
  %s  = Sub %m, imm SUB          where SUB ∈ { 0, MASK }
  %z  = X86Flag.ZF %s
  %r  = R{KIND} %z               where KIND ∈ { e, ne }
} => {
  %r  = R{ (SUB == MASK) ⊕ (KIND == ne) ? CC_SET : CC_CLEAR } %f[IDX]
}

instantiate vjcc<CF, 0x1,   0, b, ae>
instantiate vjcc<PF, 0x4,   1, p, np>
instantiate vjcc<ZF, 0x40,  3, e, ne>
instantiate vjcc<SF, 0x80,  4, s, ns>
instantiate vjcc<OF, 0x800, 5, o, no>

Simple MBA Identity Rule Reduction

Tencent VM leans on trivial MBA identity rules, applied recursively, to generate larger and larger mixed boolean-arithmetic expressions. The size of the resulting expressions is intimidating; the underlying rule set is not. Below is an exhaustive list of the Tencent MBA identity rules. Defining these in an inst-combine ruleset and running optimizations to a fixed point should fully reduce Tencent MBA.

(A|B) + (A&B)        = A + B
(A^B) + (A&B)        = A | B          [from ((B^A)+(B&A)) → x|y]
(A|B) - (A&B)        = A ^ B
((A|B)^A) + A        = A | B          [also A + ((A|B)^A)]
~((~A ^ ~B) | ~A)    = A & B          [De Morgan variant]
~(((A^B) & ~B) ^ ~A) = A & B
A - (A - (A&B))      = A & B
B - (((A&B)&B) ^ B)  = A & B

((c&A) ^ A) | A      = A              [+ all operand orderings]
(A & ((A^c) | c))    = A
(((A&c) ^ A) | A)    = A
(((c|A) ^ c) | A)    = A
(((A^B)|B) & A) + B  = A + B          [((A^B)|B)=A|B, (A|B)&A=A]
(A&B) + (A|B)        = A + B

~(A - c)             = -A + (c-1)     [reported as -A, const folded]
~( <A+c gadget> )    = -A + c

Lowering

Some preparation is required before the SSA IR can be lowered back to native code. RSP was concretized to an arbitrary constant value at the start of lifting, which allows the existing optimization passes to fold up RSP modifications the same way they fold anything else. Concrete RSP values also double as a heuristic for determining whether the lifter is currently at a VMEXIT. RSP can be kept symbolic instead, but concretization is the approach the team has taken consistently across Themida, VMProtect, vxlang, Tencent VM and Denuvo.

The other thing that must be determined is how large the original stack frame of the function was. For almost every function in an AMD64 PE file there are no dynamic stack allocations — they are genuinely rare — so it is safe to assume that at sink points in the program, meaning calls and boxed instructions, RSP will reveal the original function’s stack frame size. The authors call this technique the “stack frame high water mark”, and it is precisely the property that boxed instructions hand over for free.

Once the high water mark is known, the function’s prolog and epilog can be rebuilt to properly represent the original stack frame. Special care is needed if any spilling happens in the recompiled code. Where spilling occurs, the original function’s stack frame has to be placed after the new spill space, and any references to RSP at or above the return address must be adjusted for the newly discovered size of that spill space. Handle that correctly and the result is a genuinely proper devirtualized output for any function that does not make dynamic stack allocations.

Deobfuscation Coverage Statistics

Coverage is measured per driver as the fraction of virtualized functions successfully recompiled to native code. A virtualized function is identified by its entry trampoline — an E9 jump padded with int3. After devirtualization the trampoline retargets from the .tvm0 bytecode interpreter to the recompiled .devirt code, while functions the tool could not recover still point into .tvm0. That makes the measurement mechanical rather than a matter of judgement: count where the trampolines point.

Across the four kernel drivers, 815 of 865 virtualized functions (94.2 %) were fully recovered to native code.

Limitations

Ranged for loops cause a major issue with the aggressive indirect control flow optimization approach. Take a loop such as for (int i = 0; i < 10; i++). When it is virtualized, the condition i < 10 becomes a VJCC. But i = 0 at that point, so the optimizations fold the VJCC down to a single branch destination — the loop back edge. Since the back edge has already been lifted, lifting stops and an infinite loop is produced. Relifting back edges may be required, symbolizing everything except the VIP.

This is a good illustration of the tension at the heart of the approach: the same constant propagation that dissolves the bytecode also dissolves loop conditions that were never meant to be constant. Aggressiveness buys coverage on straight-line code and costs it on loops, which is a reasonable guess at where the missing 5.8% in the coverage table is concentrated.

Outlook: Why Classical VM Obfuscation Is Fading

The authors close by restating the belief they opened with: classical virtual machine obfuscation is not strong in the face of guided symbolic evaluation. What they consider demonstrated here is that simple compiler optimization passes — constant promotion, constant folding, and a handful of trivial MBA reduction rules — are sufficient to simplify Tencent VM. Nothing exotic was required.

As AI-assisted static deobfuscation continues to improve, they expect more virtual machine based obfuscators to fold away in the same manner. An upcoming article is promised covering the full static devirtualization of Denuvo anti-tamper and anti-cheat — by their description one of the most heavily attacked pieces of software, second only to anticheats themselves.

They also note that for their own product, CodeDefender, they have taken specific care to directly hinder lifting-based attacks and guided symbolic evaluation. Their argument is that analysing hardened targets is exactly how an obfuscation vendor identifies and avoids common pitfalls that would otherwise be overlooked. Readers interested in the product or consulting are pointed to the contact address on the original page.

Key Takeaways

• Tencent VM is structurally close to VMProtect. The original entry jumps into .tvm, all GPRs and EFLAGS are staged into a VM context, and execution threads through a shared dispatch loop — a design whose weaknesses are already well mapped.
• Boxed instructions are the load-bearing weakness. Because the VM cannot model all of AMD64, it must periodically restore real guest state and execute natively. Each of those points is a guaranteed leak of true machine state and the basis of the stack frame high water mark technique.
• CET support costs the obfuscator nothing and buys it nothing defensively. The RDSSPQ hint-NOP feature check plus INCSSPQ 2 is a clean solution to unmatched dispatcher calls, but it is also a distinctive, easily fingerprinted signature.
• SEH compatibility required real engineering. A phantom unwind function reached through [RBP+0], a UWOP_SET_FPREG on RBP, and a handler that stages guest non-volatiles into a reserved save area let the Windows unwinder reconstruct guest context transparently.
• Guided symbolic execution needs only obfuscation-specific hints. Following calls with a trailing int3, promoting bytecode loads to constants under careful scoping, and dynamically locating the VIP by finding the last store of a .tvm0 pointer are enough to drive the lifter.
• The MBA layer is shallow. Sixteen identity rules run to a fixed point fully reduce the expressions, no matter how large recursive application has made them.
• 94.2% coverage across 865 functions in four production kernel drivers — with the residual failures concentrated in loop constructs, where aggressive constant folding collapses a VJCC to a single back edge.

Defensive Recommendations

• Do not treat VM obfuscation as a security boundary. Anything that must not be recovered — keys, server-authoritative logic, licensing decisions — belongs behind a real trust boundary, not behind a bytecode interpreter that a lifting framework can fold away in an afternoon.
• Minimize boxed instruction density. Every native escape is a materialization of true guest state and a sink point an attacker can anchor on. Model as much of the instruction subset as is practical inside the VM and cluster the unavoidable escapes.
• Break the constant-folding chain. The entire attack rests on promoting bytecode loads to constants. Bytecode whose decryption depends on genuinely runtime-derived values — not values recoverable from the static image — forces the symbolic evaluator to keep it symbolic.
• Do not rely on MBA depth alone. Recursive application of a small identity set produces large expressions with a tiny rule closure. If MBA is part of the design, the rule set itself needs to be broad and non-canonical, not deep applications of sixteen textbook identities.
• Assume your VIP is discoverable. Heuristics that locate the VIP by scanning for stores of bytecode-section pointers into the context structure defeat hardcoded-offset assumptions. Diversify context layout per build if VIP concealment matters.
• Fingerprint the construct defensively. Defenders and analysts triaging unknown drivers can hunt for the pattern directly: an E9 entry trampoline padded with int3, calls immediately followed by int3, an oversized single .pdata range with a language-specific handler, and a RDSSPQ/INCSSPQ pair.
• Watch the unwind metadata. A never-executed function carrying unwind info for nine non-volatile registers, referenced through a frame-pointer slot, is anomalous enough to be a detection signal in its own right during driver review.
• Validate vendor obfuscation claims empirically. If a third party can hand you 94.2% of a protected driver back in clean native form, the marketing claim about the protection layer is not the one that matters — commission an independent lifting assessment before relying on it.

Conclusion

The Tencent VM is a competently built piece of engineering — the CET handling is neat, and the phantom-unwind-info trick that makes SEH work transparently through a virtual machine is genuinely clever. But competence in the runtime details does not translate into resistance against an attacker with a lifting and recompilation framework. The bytecode becomes a constant, the constants fold, the MBA reduces against a sixteen-rule closure, the virtualized branches pattern-match back into native JCCs, and the boxed instructions hand over the stack frame size on request. What comes out the other end is 815 of 865 functions in clean native form. For anyone weighing whether a VM obfuscator is the right place to invest protection budget, this article is a fairly direct answer.

Original text: “Static Devirtualization of Tencent VM” — author not clearly listed (site: Aftermath Labs), July 31, 2026.