A Capture-the-Flag or “CTF” is a cybersecurity competition designed to test and sharpen security skills through hands-on challenges that simulate real-world situations. MetaCTF offers training in eight different categories: Binary Exploitation, Cryptography, Web Exploitation, Forensics, Reconnaissance, Reverse Engineering, CyberRange, and Other / Miscellaneous. This blog is part of a series that explains what each category covers, why the subject is important, and how to approach these problems.
What is binary exploitation?
Binary exploitation is a technique used by hackers to make an application do things it was not designed to do. Usually, hackers manipulate the app by corrupting its memory.
When performing a binary exploitation, hackers first figure out how the app works using tools like disassemblers and debuggers. Once they understand the workflows used by the application, they will look for ways to provide corrupt inputs that either crash the app or enable execution control in ways unintended by the app’s creators.
For example, a hacker may want to escalate their user privileges to an admin level in order to bypass restrictions on how they can use the app. If the hacker discovers that the workflow that controls user permission is vulnerable to buffer overflow (a common type of binary exploitation), the hacker could input more data than the app expects, thereby sending the app’s execution paths in unexpected directions. This exploitation could enable the hacker to surreptitiously escalate their privileges by altering the execution path of the application.
How to solve a binary exploitation CTF challenge
First, look for the hint in the CTF instructions. If the title is “Stack-Based Buffer Overflow,” that is a good hint that you can solve the challenge by finding a way to use a buffer overflow on the stack.
Next, figure out how the app works. If you receive the source code as part of the instructions, start reading that code. If you don’t get the source code, you can perform dynamic analysis in your debugger and watch the execution step-by-step. Or you can perform static analysis by loading the application into disassemblers and other reverse engineering tools, which will produce a control flow graph and assembly code that you can use to understand the app’s workflows.
Finally, once you understand how the app works, look for ways to apply the hint. If the CTF suggests you will need to use a buffer overflow, look for places in the app where providing too much data may violate the programmer’s assumptions. For example, if the password field is limited to 10 characters, see what happens if you submit a password with more than 10 characters. Keep trying until you get the app to react in unexpected ways.
Resources to learn more about binary exploitation
- LiveOverflow’s YouTube series on binary exploitation
- CTF101.org’s Binary Exploitation overview
- CERN Computer Secuirty’s common vulnerabilities guide for C programmers
Want to learn more about cybersecurity training options for your team? Schedule a demo here or email us at contact@metactf.com.