Brainstorming != Security Analysis

I mentioned a community meeting I recently attended to some friends, and they were astonished to learn that not only did I think our brainstorming session was productive, I actually suggested it. Apparently, I have ranted so much about brainstorming that my friends think I disapprove of brainstorming as a concept. Whoops. :) Let me explain.

Usually groups use brainstorming to generate a bunch of ideas about some topic. Brainstorming (especially the round-robin technique we were using at the community meeting I described) is a great way to ensure that everybody's point of view is represented. It's all about inclusion, suspending judgment, listening, and other touchy-feely stuff that is very important in a community, especially one that is just forming. But as Wikipedia tactfully puts it, "researchers have not found evidence of its effectiveness for enhancing either quantity or quality of ideas generated. … [B]rainstorming groups are little more effective than other types of groups, and they are actually less effective than individuals working independently." In my opinion, the main reason to use it is to improve the team's soft skills, understanding of each other, morale, or similar, which is exactly what we needed at the community meeting.

And it should make a good security analysis technique, right? Hah. Having a happy, well-adjusted analysis team that works well together is desirable, but hardly ensures decent results. Likewise, even if brainstorming were good for generating lots of ideas, having a lot of ideas (on any topic) is not a major contributing factor for good security analysis results. Yet many current security analysis techniques have a step (e.g. "now we all get together & think up everything that could possibly go wrong") that looks a lot like brainstorming. What is going on here?

For starters, most consumers of security analysis results (e.g. non-security developers or management folks) can count just fine, but don't have the skills or knowledge to tell the difference between high and low quality analysis. Therefore, they like quantity. Analysis consumers will feel more satisfied and comfortable when analysis investigates and finds more issues. This is not entirely unreasonable; my experience indicates that even for a small system, there are usually quite a few things to investigate. If analysis only investigates 3 potential issues, I expect low quality analysis. In the frequent situation in which the analyst doesn't have a lot of security analysis experience (e.g. the do-it-yourself model many security development lifecycles suggest) but knows that investigating 3 issues is not enough, brainstorming starts to look like a good technique for identifying potential issues: its stated goal is to increase quantity of ideas, and quantity of ideas is what seems to be missing. Brainstorming is so well-accepted as an idea generation technique that it rarely occurs to anyone to investigate its effectiveness. (For the record, what is actually missing is coverage. Quantity of potential issues is a secondary indicator of coverage; quantity of found issues is a secondary indicator of a combination of coverage, analysis quality, granularity of analysis or reporting, target quality and many other interrelated factors.)

Next, most experienced security analysts can't accurately explain what they are doing when they come up with a list of potential issues to investigate. When an inexperienced security analyst asks, the more experienced guy will frequently say something like "I start by making a big list of possible stuff to look into". He might even use the word brainstorming. When the inexperienced analyst looks at the output, it will look like brainstorming output. The trick is that brainstorming is not what he's doing. What the experienced analyst is actually doing is very similar to what Benjamin Kovitz (Practical Software Requirements) explains engineers do instead of the functional decomposition they appear to do: matching what he already knows to the target at hand, and writing a list of potential issues he already knows about for this or similar targets . If you get a group of experienced analysts together, they will do what looks like brainstorming and come up with a great list, each of them unconsciously using her own experience.

What happens when an inexperienced analyst tries brainstorming for security analysis purposes? Let's say he gets a group together, and they come up with some ideas. Frequently these days, the ideas are couched as a threat model. One such threat model I vividly recall receiving (as a guide to the code review and penetration testing I was supposed to do for a client) had one (1) threat. When I asked the security-inexperienced client about it, I found the development team had brainstormed a list of threats, but the application was so small that this was the only idea they had come up with. It's true, it was a small application with limited security requirements; if I recall correctly, I came up with only 2 dozen or so threats (maybe 50ish variations for my colleague and I to investigate) when I did a brief, more systematic analysis. This is an extreme example of intelligent but inexperienced analysts applying brainstorming in good faith and getting lousy analysis results for their trouble. More typical results in my experience are that inexperienced analysts think of around 5-15% of what experienced analysts investigate, with most threat models by inexperienced authors containing 5-12 potential issues to investigate. (These are approximations based on experience, not hard statistics.)

Now imagine that the inexperienced analyst looks at the 1 potential issue he got out of the initial brainstorming session, and thinks to himself, "hmm, something went wrong; better get some help". What does he do? Calls his more experienced buddy & has the more experienced buddy facilitate a brainstorming session. I've been that buddy and talked to a lot of other more experienced buddies, and the brainstorming session facilitated by the more experienced analyst is usually pretty successful. Why? It's not brainstorming either. The experienced analyst is asking questions that will guide participants to realize the potential security issues for this target, and helping them fill in the blanks (again based on her own experience) when participants miss something.

After a brainstorming meeting, most participants will express great satisfaction with the meeting, the process, and the results. This is great, but when interpreting their happiness, remember that suspending judgment is part of the brainstorming process and a morale boost is one of the effects. Also keep in mind that people who suggest that brainstorming is not an appropriate technique to use are frequently accused of having a bad attitude, not being team players, and so on, and may therefore be keeping their mouths shut. Either participants are experienced and/or knowledgeable, in which case they unconsciously used their experience and knowledge to get good results, or they are not, in which case they usually can't tell how good their results are (yet; some participants will notice the low quality as analysis proceeds). My experience has been that by the time a brainstorming-based security analysis completes, most participants have started to question the quality of results, the ROI, or both. Unfortunately, what's done next time is often based on familiarity and the emotions people expressed during the process ("Last time I led a group through this they were very happy with the results."), rather than costs and benefits calculated at the end.

Brainstorming is NOT a security analysis technique. Any security analysis technique that looks like brainstorming is suspect. Either it doesn't work but nobody can (or will) tell, or something else is going on behind the scenes. Find out which is happening for you, then skip the brainstorming and head straight for the secret sauce.

--Brenda