BitsCTF VM Challenge (El Diablo) Writeup

Initial Analysis

First Run

When we first executed the binary, we were greeted with:

$ ./challenge
Welcome my DRM-protected application!
To sucessfully get in, you must present a valid license file.
Reverse engineer this binary to figure out the license format, and get the flag. :)
Good luck!

Usage: ./challenge <license file path>

This confirmed the binary expects a license file as input. We also noticed the binary was packed with UPX, which is a common packer that can be easily unpacked.

Static Analysis Setup

I loaded the binary into IDA Pro and discovered:

• Binary Type: Linux x64 ELF (originally UPX packed, I unpacked it using upx -d challenge)
• File Size: ~60KB after unpacking
• Entry Point: Standard _start → __libc_start_main → main

Key Observations

1. License File Required: The binary expects a license file as command-line argument
2. License Format: LICENSE-<hex_encoded_bytes> (e.g LICENSE-deadbeef)
3. Expected Behavior: Should print a flag when given the correct license

Understanding the Protection Mechanisms

Anti-Debug Protection (sub_2E82)

I analyzed function sub_2E82 at address 0x2E82:

1
int sub_2E82() {
2
    int var_1 = 0;  // Initialized to 0
3

4
    // Check 1: ptrace detection
5
    if (ptrace(PTRACE_TRACEME, 0, 1, 0) == -1) {
6
        var_1 = 1;  // Set flag on detection
7
    }
8

9
    // Check 2: TracerPid check in /proc/self/status
10
    if (check_tracer_pid()) {
11
        var_1 = 1;
12
    }
13

14
    // Check 3: Timing check with clock_gettime
15
    if (timing_check()) {
16
        var_1 = 1;
17
    }
18

19
    // Check 4: Parent PID check
20
    if (getppid() == 1) {  // Not debugged if parent is init
21
        var_1 = 1;
22
    }
23

24
    return var_1;  // Returns 0 if clean, 1 if debugger detected
25
}

Critical Discovery: The main function expects sub_2E82 to return 0 to continue execution:

1
int main(int argc, char **argv) {
2
    if (sub_2E82() != 0) {  // Inverted logic!
3
        puts("DEBUGGER DETECTED!");
4
        return -1;
5
    }
6
    // Continue to VM initialization...
7
}

This is inverted logic - the function returns 0 when clean, but sets var_1 = 1 when debuggers are detected.

VM-Based Obfuscation

After the anti-debug check, execution reaches an unusual instruction:

1
0x31D1: ud2    ; Undefined instruction - triggers SIGILL

This is intentional! The binary has a custom SIGILL signal handler that:

1. Catches the undefined instruction exception
2. Decrypts VM bytecode
3. Initializes the virtual machine
4. Resumes execution

Bytecode Encryption

The VM bytecode is stored encrypted at address 0xBB00:

• Size: 928 bytes
• Encryption: Custom AES-CTR implementation
• Decryption happens in: sub_3248 (called by SIGILL handler)
• Destination: Bytecode is decrypted to 0xC2E0 in memory

Bypassing Anti-Debug

Initial Attempts (Failed)

Attempt 1: Patch return value

1
# Tried patching the ret instruction to always return 0
2
patch_bytes(0x2F7C, b'\x31\xC0\xC3')  # xor eax, eax; ret

Failed: This broke the stack frame and caused segfault

Attempt 2: Patch jump conditions

1
# Tried inverting jz to jnz
2
patch_bytes(0x2EA9, b'\x75')  # Change jz to jnz

Failed: Only fixed one check, others still detected debugger

Attempt 3: Force return 1

1
# Tried to make it return 1 thinking that's "success"
2
patch_bytes(0x2F7A, b'\xB8\x01\x00\x00\x00')  # mov eax, 1

Failed: This made it always detect debugger

Successful Approach

We realized the key was to prevent var_1 from ever being set to 1. Each detection sets it with:

1
mov     [rbp+var_1], 1    ; C6 45 FF 01

Solution: NOP out these assignments!

1
def patch_anti_debug(binary_path, output_path):
2
    with open(binary_path, 'rb') as f:
3
        data = bytearray(f.read())
4

5
    # Patch locations where var_1 is set to 1
6
    patches = [
7
        0x2EC6,  # After ptrace check
8
        0x2EF3,  # After TracerPid check
9
        0x2F19,  # After timing check
10
        0x2F3F   # After parent PID check
11
    ]
12

13
    for addr in patches:
14
        # Replace "mov [rbp+var_1], 1" with NOPs
15
        data[addr:addr+4] = b'\x90\x90\x90\x90'
16

17
    with open(output_path, 'wb') as f:
18
        f.write(data)

This forces sub_2E82 to always return 0, allowing execution to continue!

Testing the Patch

$ chmod +x challenge_correct
$ echo "LICENSE-0000000000000000" > /tmp/test.txt
$ ./challenge_correct /tmp/test.txt
[i] loaded license file
processing... please wait...
[i] running program...
The flag lies here somewhere...
BITSCTF{<garbled output>}

Success! The binary now executes past anti-debug and reaches the VM.

VM Initialization and Bytecode Extraction

Understanding SIGILL Handler Chain

After bypassing anti-debug, execution hits ud2 at 0x31D1. The handler chain:

1. Constructor sub_33C9 (.init_array[1]): Registers SIGILL handler
2. Handler sub_3379: Catches SIGILL signal
3. Decryption sub_3248: Decrypts bytecode from 0xBB00 to 0xC2E0

The decryption key comes from qword_C290, which must be 0 for correct decryption.

Extracting Real Bytecode

Since bytecode is only decrypted at runtime, we needed to extract it:

$ gdb ./challenge_correct
(gdb) handle SIGILL pass noprint nostop  # Let handler run
(gdb) break *0x3AB4  # Break after memcpy in VM creation
(gdb) run /tmp/test.txt
(gdb) x/928bx $rsi  # Examine bytecode location
(gdb) dump binary memory /tmp/vm_real.bin $rsi $rsi+928

We extracted 928 bytes of decrypted VM bytecode.

Analyzing VM Bytecode

1
with open('/tmp/vm_real.bin', 'rb') as f:
2
    bytecode = f.read()
3

4
# Find key opcodes
5
for i, byte in enumerate(bytecode):
6
    if byte == 0x9A:
7
        print(f"GET_LICENSE_BYTE at offset {i}")
8
    elif byte == 0x9C:
9
        print(f"PRINT_FLAG_CHAR at offset {i}")

Key Opcodes:

• 0x9A (GET_LICENSE_BYTE): Reads a byte from the license file into a register
• 0x9C (PRINT_FLAG_CHAR): Prints a character from a register

VM Execution Model

The VM operates on 10 registers (each 16 bytes):

• Structure: value (8 bytes) + flags (8 bytes)
• Operations: Load license, XOR, arithmetic, conditional jumps
• Output: Characters printed via opcode 0x9C

Flag Extraction Strategy

Initial Testing

I then wrote a test license of all zeros:

$ echo "LICENSE-0000000000000000000000000000000000000000000000000000000000000000" > /tmp/test.txt
$ PRINT_FLAG_CHAR=1 ./challenge_patched /tmp/test.txt
BITSCTF{y3r_by_l�E3r_y0u_u�N4v3l_my_�cr375}

Observations:

1. Prefix BITSCTF{ appears correctly
2. Many characters are garbled (non printable)
3. Some readable patterns: y3r_by_l, 3r_y0u_u, 4v3l_my_, cr375}

Byte-by-Byte Solving

I then created a solver that brute-forces each license byte:

1
def solve_position(lic_bytes, position, target_char):
2
    """Find license byte that produces target_char at position"""
3
    for byte_val in range(256):
4
        test_lic = list(lic_bytes)
5
        test_lic[position] = byte_val
6

7
        output = test_license(test_lic)
8

9
        if len(output) > position and output[position] == ord(target_char):
10
            return byte_val
11

12
    return None

Solving the Prefix

Target: BITSCTF{

1
Position 0 → 'B' (66): Found license[0] = 0x99
2
Position 1 → 'I' (73): Found license[1] = 0xf5
3
Position 2 → 'T' (84): Found license[2] = 0x67
4
Position 3 → 'S' (83): Found license[3] = 0x11
5
Position 4 → 'C' (67): Found license[4] = 0x24
6
Position 5 → 'T' (84): Found license[5] = 0xd5
7
Position 6 → 'F' (70): Found license[6] = 0x20
8
Position 7 → '{' (123): Found license[7] = 0xd5

License so far: 99f5671124d520d5

Discovering Position Variability

We made a critical discovery that not all positions are variable

1
# Test if position changes with different license bytes
2
for lic_byte in range(256):
3
    test_lic[10] = lic_byte
4
    output = test_license(test_lic)
5
    print(f"lic[10]={lic_byte:02x} → output[10]='{chr(output[10])}'")
6

7
# Result: ALL values produce 'y' at position 10!

Analysis:

• Positions 10-17: CONSTANT (always same character regardless of license)
• Positions 20-27: CONSTANT
• Positions 30-31: CONSTANT
• Only positions 8, 9, 18, 19, 28, 29 vary with license!

This told us the VM has conditional logic that only uses certain license bytes.

Identifying the Pattern

1
BITSCTF{lay3r_by_l??3r_y0u_u???4v3l_my_??cr375}
2
         ^^^^^^^^^    ^^^^^^^^^     ^^^^^^^^
3
         "layer by l" "er you u"    "el my secrets"

Solving Variable Positions

Position 8: First character after {

1
# Target: 'l' (layer)
2
Found: license[8] = 0xf6 produces 'l'

Position 9: Second character after {

1
# Target: 'a' (layer)
2
Found: license[9] = 0x69 produces 'a'
3
Output: BITSCTF{lay3r_by_l4,3r_y0u_un'4v3l_my_5fcr375}

Wait, position 19 shows ’,’ but we want ‘y’ for “layer”!

Position 9 (retry): Testing more values

1
# We need 'y' at position 19, not ','
2
# Test all bytes for position 9 and check position 19!
3
for byte_val in range(256):
4
    test_lic[9] = byte_val
5
    output = test_license(test_lic)
6
    if output[19] == ord('y'):
7
        print(f"Found: {byte_val:02x}")
8
        break
9

10
# Found: license[9] = 0x3c produces 'y' at position 19!

Understanding License-to-Output Mapping

The VM doesn’t have a 1-to-1 mapping. Some license bytes affect multiple output positions:

This explains why changing license[9] affected both position 9 and 19!

Final Solution

The Flag

$ echo "LICENSE-99f5671124d520d5f63c00000000000000000000000000000000000000000000" > /tmp/final.txt
$ PRINT_FLAG_CHAR=1 ./challenge_patched /tmp/final.txt
BITSCTF{l4y3r_by_l4y3r_y0u_unr4v3l_my_53cr375}

Decodes to: “layer by layer you unravel my secrets” well it wasn’t exactly the same but close enough ig xD

The License

1
LICENSE-99f5671124d520d5f63c00000000000000000000000000000000000000000000

Thank you for reading this long aah writeup! I hope it provided insight into reversing VM-based obfuscation and bypassing anti-debugging techniques.