CSE4303 Introduction to Computer Security (Lecture 19)

Due to lack of my attention, this lecture note is generated by AI to create continuations of the previous lecture note. I kept this warning because the note was created by AI.

Fuzz testing

Software testing overview

Software testing in a box

White box testing
- Often “unit” testing, static analysis, code coverage
- Tests internal parts of the program individually
- You choose what gets tested
- You often test your own programs based on how you “know” they should work
Black box testing
- Assume you don’t know/care how program internals work
- Tests what a program does, not how it does it
- Provide input, ensure output is correct and no errors
Both are important!

Introduction to fuzz testing

What is fuzz testing?

Form of black box (usually) testing
Generate bad, unexpected, or random input to try to cause an error
Tries to find bugs/errors as opposed to testing for correctness
A fuzzer automates fuzz testing
- Range from one-line scripts
- … to “intelligent” and advanced programs like afl-fuzz
- All automatically generate input in an attempt to cause an error
Sometimes used to point developers to areas of the program that need attention instead of explicitly finding bugs

Types of fuzzing

Mutative vs. Generative
- Mutative takes a starting input and makes changes (mutations) to it
- Generative takes a specification and generates input based on it
Dumb vs. “Smart”
- Dumb makes changes without knowledge of input structure
- Smart knows at least something about input structure
- Generative fuzzers can’t be completely dumb
Binary-only (black box) vs. binary and source (white box)

Pros and Cons of fuzzing

Pros:

Great at finding memory/safety/error handling bugs
Can be fully automated and easily run
Finds bugs usually missed by humans

Cons:

Hard to do well
Only finds certain classes of bugs
Can take a long time
- If input is n bytes long, there are 256^n possible inputs!
Doesn’t test program correctness