crypto ctf guide
how to approach cryptography challenges.
classical ciphers · RSA · AES · XOR · hashes · tools
two types of crypto challenges
most CTF crypto challenges fall into one of two buckets:
- classical / historical ciphers — Caesar, Vigenere, substitution, transposition. broken by frequency analysis. CyberChef handles most of these.
- modern crypto misuse— RSA, AES, ECC — implemented with a flaw. the math is correct but there's a weak key, bad mode, or reuse that makes it breakable. you need to identify the flaw, not break the algorithm.
reading a crypto challenge
- 1.identify what encryption scheme is used — the challenge script usually tells you.
- 2.look for what's being reused: same key for multiple messages? same IV? same nonce? reuse is almost always the vulnerability.
- 3.look for small parameters: small e in RSA, short key length, small prime factors.
- 4.if you're given source code, read it carefully for logic errors in the encryption or key generation.
- 5.google the scheme + "ctf attack" or "ctf exploit" — most CTF crypto vulnerabilities have named attacks.
common attack patterns by scheme
- RSA — e=3, small messagem³ < n means c = m³ in the integers. take the integer cube root of c to get m directly.
- RSA — common modulussame n, different e values for the same message. use extended Euclidean algorithm on (e1, e2) to recover m.
- RSA — Wiener's attackd is suspiciously small. use continued fractions on e/n to recover d.
- RSA — factor nn is small enough for factordb.com or yafu. always check before doing anything else.
- AES ECBidentical plaintext blocks → identical ciphertext blocks. detect by looking for repeated 16-byte blocks. exploit with chosen plaintext.
- AES CBC — padding oracleserver leaks whether padding is valid. decrypt byte-by-byte by manipulating the IV/ciphertext and watching the oracle's response.
- XOR — single byte keytry all 256 possible keys and score by English letter frequency. Python: max(range(256), key=lambda k: score(bytes([b^k for b in ct])))
- XOR — repeating keyfind key length via index of coincidence or Hamming distance between chunks. then solve each key byte position independently.
useful tools
- CyberChef — decode, encrypt, transform. fast for classical ciphers and encoding.
- RsaCtfTool — automated RSA attacks. throws all known attacks at an RSA public key.
- factordb.com — check if n has already been factored. always try this first for RSA challenges.
- SageMath — number theory, polynomial arithmetic, elliptic curves. essential for advanced crypto.
- hashcat / john — hash cracking. use with rockyou.txt wordlist for common hash challenges.
- pycryptodome — Python crypto library. use for scripting attacks once you know the approach.