how to approach binary exploitation challenges.

first steps for any pwn challenge

1. 1.run checksec on the binary first. note which protections are enabled: NX (no executable stack), PIE (position-independent executable), stack canary, RELRO, FORTIFY.
2. 2.run file on the binary — 32-bit or 64-bit? ELF or PE? stripped or not? this determines calling conventions and exploit approach.
3. 3.run strings on the binary. look for flag-like strings, function names, hard-coded paths, or interesting messages.
4. 4.run the binary with ltrace or strace to see which library calls and syscalls it makes without reversing the assembly.
5. 5.open in Ghidra or GDB. find main(), the vulnerable function, and understand what input the program reads and how it processes it.

match the protection to your exploit path

• no NX, no canaryclassic stack smash — overflow the buffer, overwrite return address with pointer to shellcode you placed on the stack.
• NX enabled, no PIEret2libc — leak not needed if no ASLR. find plt/got addresses statically. ROP to system("/bin/sh").
• NX + PIE enabledneed to leak a runtime address first. find a format string or partial overwrite to leak libc/binary base, then calculate offsets.
• stack canaryneed to leak the canary value before overwriting it, or find an arbitrary write that skips the canary entirely.
• full RELROGOT is read-only — can't overwrite GOT entries. target __malloc_hook, __free_hook, or one_gadget in libc instead.

common challenge types and approach

• ret2wina win() function exists in the binary that prints the flag. find its address, overflow the buffer to overwrite the return address with it. offset = cyclic pattern in GDB.
• ret2libcno win function — ROP to call system("/bin/sh"). find a gadget to set RDI (64-bit) or push the argument (32-bit), then call system@plt or system via libc offset.
• ROP chainbuild a chain of small code snippets ending in ret (gadgets) to set registers and call functions. use ROPgadget or ropper to find available gadgets.
• format stringprintf(input) with no format string — %p leaks stack values, %s dereferences pointers, %n writes to memory. calculate the offset to the format string's own position on the stack first.
• heap exploitationfree a chunk then use it again (UAF), or corrupt heap metadata. modern glibc uses tcache — double-free corrupts fd pointer, enabling allocation over arbitrary address.

useful tools

• pwntools — Python exploit scripting library. connects to remote, crafts payloads, handles encoding. nearly every pwn solve uses it.
• GDB + pwndbg — dynamic analysis. set breakpoints, inspect registers and stack, step through assembly. pwndbg adds heap inspection and pattern commands.
• Ghidra — free decompiler from NSA. turns assembly into readable C-like pseudocode. essential for understanding binary logic without source.
• ROPgadget / ropper — find ROP gadgets in the binary. filter by instruction sequence to build your chain.
• checksec — audit binary protections in one command. tells you NX, PIE, canary, RELRO status.
• one_gadget — find single gadgets in libc that call execve("/bin/sh") with specific register constraints. faster than a full ROP chain if constraints match.