By Jacob Finkelman: Rust Foundation Security in Residence (SEIR)
Over the past few months I have been the AI Security Engineer in residence at the Rust Foundation, which means I’ve spent my time looking at a truly staggering number of AI reports across a very large number of projects. With a decent AI Model and harness, those reports are technically reproducible. But a reproducible finding still leaves the critical question about how much anyone cares. For better – and worse – the vast majority of reports are uninteresting, at least for Rust projects. Most can be summarized as “If you use the project in a way it wasn’t intended, then you have a bad user experience.” The point of my job is to figure out which of these are worth reporting. Even after reading the documentation, it’s usually pretty hard to tell which use cases were intended.
I have often been thrilled to discover software I maintain is being used in ways I couldn’t have imagined. Free software is provided as is, you don’t have to use it the way it was intended. Sometimes an unintended use case is a wonderful feature request. On the other hand, “as is” means the maintainer is under no obligation to fix something just because someone thinks there is a bug. A healthy, hopefully pleasant, conversation is necessary to decide how best to accomplish users goals. All of this can only happen if there is some mutual understanding between the users and the maintainers about what the library is good for.
The main point
If you want better bug reports ensure your documentation has at least one sentence that answers questions like:
- What is your project good for?
- When should someone use your project?
- When should someone not use your project?
Answer as best you can, preferably in that order. Each additional sentence makes a difference to the quality of AI scans, and the ability for the operator to tell how much you care about the result. You can always add more detail when people ask questions or make reports that are confused about important details. Even just getting started makes a big difference. In other words, don’t fall into the trap of the “perfect being the enemy of the good”.
Examples
Let’s walk through an example. The other day I opened my AI harness of choice and it reported a new bug entitled “Diff diagnostics exhibit quadratic CPU growth on repeated lines”. I checked the test case it provided and it was technically correct. If the input strings only differed in their last line, then comparing them took quadratic time. The documentation goes into some detail about how to use the project. It does a pretty good job of answering the first question. The maintainer is a friend of mine, so I grabbed his attention away from other important work to review my security findings. He pointed out that the algorithmic complexity of each operation is not guaranteed, he did not consider it a security concern. If that had been in the documentation, I don’t think the security focused AI harness would have reported the problem to me. Even if it had, having seen that documentation, I would not have escalated it so dramatically to him.
Another example involved a library that described itself only as “ergonomic inter process communication”. It had a bug where receiving a malicious message sent over the channel triggered a panic. Inter process communication is used for many things, which one was this library intended to be used for? Some uses assume both peers are trusted, others are intended to be used in cooperation with OS Sandboxing tools to work with untrusted code. Knowing whether the library supported communication with untrusted peers would have helped me decide what to ask the maintainer to fix and what to warn users about.
Being more precise
Sometimes it’s hard to summarize the answers into one sentence. Perhaps your product has several different kinds of relevant users. Similarly, those users might want to use your product for different reasons and in different ways. Or you don’t know how well you can support a particular use case, and would like to hear from users before you decide. You are not the first to have these problems. There’s a whole field called “Threat Modeling” that has its own set of jargon and norms for precisely answering these questions, even in complicated situations. If picking up some of this jargon makes it easier to describe your project, use it. If not, then a plain description will do just fine.
Luckily AIs are pretty good at going from your source code and documentation to produce formal Threat Models. This means that if you only answer the three basic questions in a sentence or two, AI tooling wanting a formal threat model in some particular format can generate an approximation for itself. For example, The threat model skill is one of the key tools that make Scrutineer such an effective scanning harness.
An aside on owning the threat model
This also means that if you would like to have control over the formal Threat Model used for your project, you can get a pretty good draft by just asking an AI. You may have to fix some things that are obviously wrong, or improve your documentation and try again, but that’s not failure. This is an iterative process. You can always improve when you see how people and tools are misinterpreting the draft you currently have. Alternatively, people have worked very hard on skills to teach AIs how to do this kind of iteration.
Maintaining a fully formal threat model can be a lot of work. Every new feature can change how your code is intended to be used. Every novel way for attackers to hack programs can change what defenses are needed. Lots of projects find that unsustainable. So maintain documentation at whatever level of abstraction or specificity is sustainable for your project, and let the AIs formalize or point out gaps on an as needed basis.
Citing my sources
Many people have had this insight before, but they tend to be people who are quite comfortable with threat models and their jargon. “Not a Security Issue” documented the effectiveness of curl’s threat model for the efficacy of AI tools. “Why Most Lodash Vulnerability Reports Get Rejected” describes the important role of a threat model. “We have to change the rules of security” advocates for changing your threat model to reduce the number of bugs that get the full embargoed security treatment. The “Maintainer’s Guide to GHSAs” points out the complementary advantage that a clear threat model makes it easier for the maintainer once they receive a report.
Conclusion
Handling potential security findings hinges on a healthy conversation about what a project is supposed to be used for. That conversation involves a lot of context that is often missing from documentation. Recording that context can make future conversations significantly easier. A few sentences won’t resolve every argument, but it gives a starting point for the next conversation to cover new ground. So please go document what your project is good for, when should someone use your project, and when should someone not use your project. Do so now, in your own words, without waiting for all the specialist terminology of threat modeling, so that we can have more productive conversations.
About the Author
Jacob Finkelman is the AI Security Engineer in Residence at the Rust Foundation. He has been a member of the Rust Cargo Team since 2018 and became its Co Lead in 2026.
