The interesting thing to me is the stark difference between this and golang's approach.
With golang, you can run fuzzing as simply as you run tests, which means that it's trivial to target specific parts of your application or library. It obsoletes so much of those techniques.
I'm quite curious of techniques to guide more the fuzzing. It seems like the best you can do is provide a seed corpus and hope for the best.
some fuzzing tools (libFuzzer for example) leverage LLVM's intermediate representation to provide code-coverage metrics that they feed back into their fuzzing algorithms, increasing test coverage
LibFuzzer is packaged with clang, so there is no additional installation [0]. You just have to provide an entry function and link it with a command-line flag. However, since C and C++ lack reflection you have to work with raw bytes as input.
LibFuzzer has the option to provide callbacks that customize mutation, which can help with obtaining coverage.
I think it would go the other way where you use coverage to guide reinforcement. Crank the temperature up to increase variation and you would probably produce a model that could approximate the file format you were targeting.
> Fuzzing is often a special case of genetic algorithms
Yes, that was sort of why I thought RL guided fuzzing could work, and possibly better. Also, for things like XSS fuzzing (which I have a little experience in), it is possible for an experienced attacker to intelligibly guide the fuzzing to a payload, which theoretically could be mimicked through RL.
There wasn't really anything novel in the proposal, it was just for a graduate cyber-security course, and one of the deliverables was a project proposal for something related. There were already some existing works that time (around 2-3 years ago) where people tried combining RL with fuzzing, and I just mish-mashed some ideas together so I could hand in something.
My main concern at the time however was that with fuzzing the positive signal would be so rare compared to the negative signal, since most randomly fuzzed inputs would just return the same negative feedback. I wasn't sure that would be enough signal to train an RL system. I'm not quite sure what new progress has been made in the field since then.
With golang, you can run fuzzing as simply as you run tests, which means that it's trivial to target specific parts of your application or library. It obsoletes so much of those techniques.
I'm quite curious of techniques to guide more the fuzzing. It seems like the best you can do is provide a seed corpus and hope for the best.