reversing ctf hints
spoiler-free hints for reverse engineering challenges.
static analysis · decompilation · anti-debug · obfuscation · patching · VM reversing
what reversing ctf challenges look like
reversing challenges give you a binary, script, or bytecode and ask you to figure out what it does — usually to find what input produces the flag. start with file and strings to understand the binary, then open in Ghidra or IDA to decompile, and use GDB to run it and observe behavior.
common reversing challenge types
- license key / crackmefind the validation logic in the decompiled code. look for strcmp, strncmp, or custom comparison functions. patch the jump condition or extract the key directly.
- anti-debug tricksptrace(PTRACE_TRACEME) check, timing checks, IsDebuggerPresent. patch the check in the binary or use a plugin (ScyllaHide) to bypass it.
- packed / obfuscated binaryUPX-packed binaries can be unpacked with upx -d. for other packers, let the binary unpack itself in memory with GDB and dump the decrypted code.
- custom VM / interpreterthe binary implements its own instruction set. reverse the opcode handler (usually a big switch statement) to understand the VM's instructions, then disassemble the embedded bytecode.
- string encoding / XOR decodingstrings are decoded at runtime. set a breakpoint after the decode function and read the result from memory, or script the decode in Python.
- angr / symbolic executionif the validation is complex, use angr to symbolically execute the binary and find the input that reaches the "correct" branch.
- Android APK / Java bytecodeuse jadx or apktool to decompile APKs. look for flag validation in the MainActivity or a native library loaded with System.loadLibrary.
useful tools
Ghidra (free) or IDA Pro for decompilation · GDB with pwndbg for dynamic analysis · x64dbg for Windows binaries · Binary Ninja for scripting analysis · angr for symbolic execution · radare2 for scripting · jadx / apktool for Android · upx for unpacking packed binaries.
faq
Ghidra or IDA for reverse engineering challenges?+
Ghidra is free and its decompiler is strong enough for almost every CTF challenge. IDA is faster and has a larger plugin ecosystem if you own a license, but Ghidra will not hold you back in a competition. start with Ghidra and add GDB for dynamic analysis.
how do I get past anti-debugging checks?+
identify the check first — ptrace(PTRACE_TRACEME), timing deltas, or IsDebuggerPresent — then either patch the branch in the binary or defeat it at runtime with a plugin like ScyllaHide. often it is easier to neutralise the check statically in the decompiler than to fight it live.
when should I reach for symbolic execution?+
use angr when the validation logic is tangled but the success condition is clear. instead of reversing every comparison by hand, let angr explore paths and solve for the input that reaches the "correct" branch. it shines on long input-checking routines and custom encodings.