Logo

Escaping the Rusty ioctl Labyrinth

heathen heathen
March 12, 2026
3 min read
CTF rev writeup
index

DiceCTF rev_explorer Writeup

rev_explorer
Solver
h heathen
Category
Rev
Flag
dice{twisty_rusty_kernel_maze}

I’m back with another reverse engineering writeup. This time we are looking into a randomized 3D kernel maze challenge that required reverse engineering a custom Rust ioctl driver and developing a DFS solver with reset/replay capabilities to escape the labyrinth.

1. Initial artifacts and environment

  • bzImage
  • initramfs.cpio.gz
  • nc endpoint: explorer.chals.dicec.tf 1337

Connecting to nc showed a booting VM (SeaBIOS/iPXE) and eventually a shell as ctf.

/dev/challenge existed and was world RW, but direct read/write failed with Invalid argument, indicating ioctl protocol.

3. Extracting kernel ELF from bzImage

bzImage is a packed boot image, not directly pleasant to reverse. I extracted embedded compressed kernel stream and decompressed to:

  • vmlinux_from_17059.bin

String triage on vmlinux revealed:

  • drivers/misc/challenge.rs (Rust driver source path markers)
  • drivers/misc/challenge.rs:170, :266
  • dice{fake_flag_for_testing} (decoy)

Ofcourse no real flag plaintext appeared statically, so I had to dive into the code logic to find the winning conditions and how to trigger them.

4. Reversing it with IDA

I did the binary analysis in IDA on vmlinux_from_17059.bin and found several key functions in the challenge driver. I renamed them for clarity as I worked through the code:

  • challenge_init - module initialization
  • challenge_open - file open handler
  • challenge_release - cleanup on close
  • challenge_unlocked_ioctl - ioctl entry point
  • challenge_ioctl_dispatch - command router
  • challenge_build_graph_state - maze state builder
  • challenge_step_from_idx - maze traversal logic
  • challenge_read_u32_from_user - user space reader

Important globals:

  • challenge_fops
  • challenge_miscdev

Key behavior

  • open() allocates/builds per-file graph state (fresh randomized maze)
  • release() drops refcounted state
  • unlocked_ioctl() is the full user interface

5. Ioctl protocol map

Recovered from challenge_ioctl_dispatch:

  • 0x80046480 -> read field (+0x50 in state)
  • 0x80046481 -> read dim_a
  • 0x80046482 -> read dim_b
  • 0x80046483 -> read dim_c
  • 0x80046484 -> read steps
  • 0x80046485 -> read status
  • 0x80046486 -> read available-direction bitmask (6 bits)
  • 0x40046488 -> move(direction 0..5)
  • 0x6489 -> reset to start node, clear status/steps
  • 0x80406487 -> flag output path, only valid when status==1

Direction semantics are 6-neighbor 3D moves (±x,±y,±z\pm x, \pm y, \pm z equivalent mapping).

6. Graph/maze generation findings

challenge_build_graph_state constructs a per-open randomized 3D graph:

  • dimensions vary by session (observed around 8..16 each axis)
  • start/root node resettable via 0x6489
  • each node has up to 6 directional edges
  • node metadata can set terminal status on entry

Observed at runtime:

  • status=0 normal
  • status=2 common terminal trap
  • status=1 rare win state

This is why random walking often looked “stuck” in status-2-heavy behavior.

7. Runtime probing and solver development

I built minimal static syscall-only Linux x86_64 clients (no libc) and uploaded over nc shell.

Why naive DFS failed initially

After stepping into terminal nodes, move/backtrack often fails. So classic recursive backtracking breaks.

Fix

Use reset + replay:

  1. keep recorded path prefix
  2. when backtracking becomes invalid, send reset ioctl (0x6489)
  3. replay moves from root to desired depth
  4. continue DFS from there

This made traversal complete and deterministic.

8. Critical moment: proving reachability of status==1

Deterministic DFS eventually found status==1 in live runs, proving win state is reachable (not dead code).

Early attempts that called flag ioctl from deep stack sometimes segfaulted in user process.
Final reliable behavior:

  • On first status==1, immediately call 0x80406487, print output, exit.

That eliminated instability.

9. Final successful run

From remote run logs:

  • status1 reached
  • io87 rc=0 out=dice{twisty_rusty_kernel_maze}

This confirmed the full solution path worked end-to-end, escaping the labyrinth and retrieving the flag. It took me a while to write the reset/replay logic, but it was satisfying to see it succeed in the end. The key was proving that status==1 was actually reachable and not just a distraction, which required careful handling of the ioctl interactions and state management.

I took a while to publish this writeup because of tight schedules but ill be back with some interesting blogs! Thanks for reading.