Challenge of the Month: Share Your Metrics for Success in Threat Modeling
About this challenge:
Help define what success looks like in threat modeling. Let’s share insights on the key metrics that your teams use to measure the effectiveness of threat modeling practices. Your contributions will help shape best practices within the TMC community and improve our collective understanding of success in threat modeling.
How to participate:
Reply to this post with the metrics you use to evaluate success/progress/impact in your threat modeling efforts. Extra points for sharing the context—why you chose certain metrics over others, how they align with your goals, what outcomes your team aims to achieve through threat modeling.
When to enter:
Now through January 31, 2025
Recognition:
Everyone who completes this challenge will earn a Community Maven badge. Members with 7 Community Maven badges will receive a special award.
As one of the main goals of threat modeling is eliminate risk as early as possible in the SDLC process, 2 good indicators for eliminated risk are:
Number of findings identified through threat modeling (grouped by severity)
Number/Percentage of findings fixed (grouped by severity)
For example, you can track all your findings as tickets and add a special label (e.g. threat_model_finding), and use your ticketing system reporting to monitor these 2 metrics.
Parts of Success: Effectiveness (achieving the goals) Efficiency (good quality, low cost) Satisfaction
Effectiveness:
Better started than omitted.
Better completed than stuck.
Better implemented than undone.
Re 1: Define / follow / check what triggers should spawn a threat model creation or update.
Re 2: Once started, threat model can have guidance, checklists, reviews, state transitions. Progress can easily be monitored and stuck threat models identified.
One of the metrics I like to use is ‘percentage of systems within an engineering org group that have been threat modelled’.
Some approaches to threat modelling look to threat model a product that might be created by multiple teams, but if possible I like to restrict the scope of my threat models to just a system created by an individual team (Conway’s Law says these systems will combine into a whole product anyway - so I nicknamed this approach “Conway’s Threat Model”).
One benefit of this approach is that if you have an Engineering org split into groups, each with multiple teams, and you have some sort of inventory of systems these teams create, then you can report the percentage of systems that have been threat modelled per engineering org group (because a threat model only ever covers a system wholly owned by a single team in that group).
If you report that metric for each group to the head of engineering then they can easily see which of the groups that reports to them has good threat modelling coverage and which don’t, which plays nicely into each groups desire to have their metrics similar if not better than other groups. Creating that competitive angle between groups works to encourage those groups to get better threat modelling coverage of their systems
If you also expire your threat models if they haven’t been updated within a period of time (e.g. 1 year) then you get this continual effort to do regular threat modelling.
Threat Modeling Training Coverage:
Number of teams or squads who have received formal training on threat modeling. It can be tracked via internal skill development or learning management systems.
Threat Modeling Adoption:
Number of projects, changes or systems that undergo a threat modeling exercise. It can be monitored by having a task in tools like program/portfolio/project/change management systems or linking/reporting in an enterprise architecture repository.
One idea that I want to explore more in practice is to show the value of a threat model where dollars are the metric.
For example, if you have a bug bounty program you can take the issues from your threat model, then imagine them as being reported in your bug bounty program. What would each one cost in bounty payment? That total is basically your cost metric in dollars. It really adds up over dozens of models.
This idea is from OWASP AppSec Lisbon while discussing security champions programs, but basically I think the scenario can translate to threat modeling.
I completely agree, this idea is a very easy and great way to manage threat model findings. If you use something like Atlassian for tickets you can use Confluence pages to track your threat models. When you link a JIRA ticket to a confluence page you get the benefit of seeing a tickets status beside a finding on the model when you open the page.
Interesting, but i don’t think the bug bounties is what we should measuring against, the main goal of AppSec is to discover and eliminate risk from your application, which can be roughly measured by the amount of findings discovered and fixed, and In my opinion the challenge here is translating this into an actual cost, e.g. what is the cost of a critical finding in terms of:
Lost revenue, e.g. inability to land new deals, compensation, … etc
Time spent fixing
So the idea is put an estimated cost on each finding based on severity and criticality of application for e.g. and calculate the total cost, then subtract the cost of the AppSec activity (e.g. threat modeling in this case) in terms of time spent on it, cost of tools used, … etc.
Let’s say an issue is too expensive to find and fix in contrast to paying the bills when things get broken, exploited, hurt and upset … (or betting on “that will never happen”).
It is sometimes better to just do the right thing.
Not saying we can spend infinite effort and don’t have to take some of the risks (or we will go broke and never ship).
Number of risks identified and assessed as outside the risk tolerance/appetite
Number of vulnerable components identified
Number of redundant/misplaced security controls in current design
Number of security controls suggested to enhance current design to mitigate identified risks (this also translates into new system functionality/requirements)
Effectiveness (achieving the goals)
Efficiency (good quality, low cost)
Satisfaction
broken, exploited, 
hurt and 
upset … (or betting on “that will never happen”).