Binary Obfuscation that Doesn't Kill LTO: A Recap of My Thotcon Talk

At Thotcon 2025, I delved into the tension between Binary Obfuscation and Link-Time Optimization (LTO) in game security, particularly for platforms like the Nintendo Switch. While security experts appreciate the unpredictability of randomized function layouts, LTO is crucial for maintaining the spatial locality that modern CPUs, especially ARM64 architectures, rely on to minimize instruction cache misses. Blindly shuffling code can devastate performance, so we devised a method called "Apartment-Level Randomization" to balance security and efficiency.

## The Architectural Bottleneck: Instruction Cache and ARM64

Understanding the hardware is key to appreciating the risks of code shuffling. CPUs use the Instruction Cache (I-Cache) to keep pipelines full, and LTO ensures that frequently interacting functions are memory neighbors, reducing cache thrashing. On ARM64, used by the Switch, branch instructions like `B` and `BL` can span +/- 128MB, but excessive long-distance jumps from random shuffling can lead to I-Cache misses and reduced frame rates.

## The Tooling

Our research pipeline included tools like LIEF for modifying executable formats, Intel XED and Capstone for cross-platform disassembly, and an internal Intermediate Representation (IR) for applying obfuscation passes akin to compiler backends.

## The Algorithm: Step-by-Step Apartment-Level Randomization

The solution is smarter shuffling, not stopping it. We treat the binary like an apartment complex, grouping frequently interacting functions into "apartments." This involves analyzing call graphs to maintain LTO-optimized distances, defining apartments based on a short jump threshold, and ensuring performance-critical calls remain local.

### 1. Call Graph Analysis

Using graph searches, we identify function relationships and their original LTO distances, setting these as performance benchmarks. Tools like Ida Pro's Python scripting library facilitate this analysis.

### 2. Defining the "Apartment" Heuristic

Functions within a short jump threshold are grouped into the same apartment. If a callee is too far, it forms a new apartment, acknowledging the performance hit of long jumps while keeping unrelated code out of hot cache zones.

### 3. Randomizing the "City"

Apartments are randomized to disrupt static analysis and offset-based hacking, but intra-apartment function layouts can also be shuffled if performance allows.

## Visualizing the Hit: The Kcachegrind Workflow

Performance must be measured, not guessed. Using Valgrind's cachegrind, we simulate the Switch's I-Cache on ARM64 machines to assess our obfuscation's performance tax. Kcachegrind helps visualize instruction cache misses and call density, ensuring secure yet performant binaries.

## The Future: CI-Driven Optimization

Future optimizations should be CI-driven, with automated loops adjusting heuristics to maximize entropy while keeping performance degradation below 5%. By respecting hardware and leveraging tools like LIEF, Capstone, and Valgrind, we demonstrate that security and performance can coexist.