crypto ctf guide

how to approach cryptography challenges.

classical ciphers · RSA · AES · XOR · hashes · tools

two types of crypto challenges

most CTF crypto challenges fall into one of two buckets:

  • classical / historical ciphers — Caesar, Vigenere, substitution, transposition. broken by frequency analysis. CyberChef handles most of these.
  • modern crypto misuse— RSA, AES, ECC — implemented with a flaw. the math is correct but there's a weak key, bad mode, or reuse that makes it breakable. you need to identify the flaw, not break the algorithm.

reading a crypto challenge

  1. 1.identify what encryption scheme is used — the challenge script usually tells you.
  2. 2.look for what's being reused: same key for multiple messages? same IV? same nonce? reuse is almost always the vulnerability.
  3. 3.look for small parameters: small e in RSA, short key length, small prime factors.
  4. 4.if you're given source code, read it carefully for logic errors in the encryption or key generation.
  5. 5.google the scheme + "ctf attack" or "ctf exploit" — most CTF crypto vulnerabilities have named attacks.

common attack patterns by scheme

  • RSA — e=3, small messagem³ < n means c = m³ in the integers. take the integer cube root of c to get m directly.
  • RSA — common modulussame n, different e values for the same message. use extended Euclidean algorithm on (e1, e2) to recover m.
  • RSA — Wiener's attackd is suspiciously small. use continued fractions on e/n to recover d.
  • RSA — factor nn is small enough for factordb.com or yafu. always check before doing anything else.
  • AES ECBidentical plaintext blocks → identical ciphertext blocks. detect by looking for repeated 16-byte blocks. exploit with chosen plaintext.
  • AES CBC — padding oracleserver leaks whether padding is valid. decrypt byte-by-byte by manipulating the IV/ciphertext and watching the oracle's response.
  • XOR — single byte keytry all 256 possible keys and score by English letter frequency. Python: max(range(256), key=lambda k: score(bytes([b^k for b in ct])))
  • XOR — repeating keyfind key length via index of coincidence or Hamming distance between chunks. then solve each key byte position independently.

useful tools

  • CyberChef — decode, encrypt, transform. fast for classical ciphers and encoding.
  • RsaCtfTool — automated RSA attacks. throws all known attacks at an RSA public key.
  • factordb.com — check if n has already been factored. always try this first for RSA challenges.
  • SageMath — number theory, polynomial arithmetic, elliptic curves. essential for advanced crypto.
  • hashcat / john — hash cracking. use with rockyou.txt wordlist for common hash challenges.
  • pycryptodome — Python crypto library. use for scripting attacks once you know the approach.