Providing More Value in Pen Test Reports (Part 1)

So, you’ve pwned all the things, you have the keys to the kingdom, and you’ve pilfered all the data. You’ve done a great job, and now you’re done right? Not so fast.

I’ve been working in the offensive side of the infosec space for a while now, and I’ve seen a number of competitors’ reports come across my desk; some are very well put together while others are not so great. If we are all a bit honest, I’m sure we can remember a time where our own reports were pretty crappy too.

It wasn’t until a client interaction about 5-6 years ago really made me site down and consider what I was giving my clients. I had pretty much just finished a penetration test similar to the one I just described above, managing to gain access to the entire organization. I tediously documented a long report full of vulnerabilities and sent it off to the client. As usual, we had a call a few weeks after to discuss their thoughts and close out the project. The client was a bit distraught on the phone. They were happy that I was able to bring all of these vulnerabilities to their attention, but they made a few comments that stuck with me; What does this all really mean? Where do we start? Will this fix our problems and make us more secure?

Now, you might think that is pretty obvious. It means your security is not great and you should start at the beginning (duh!) and then all your problems will be solved, right? Well maybe… but there is a lot more to it. I tried to put myself in my client’s shoes, doing my best to understand what they might be thinking when they read my report. With that mindset, I sat down and read through the report myself. It wasn’t before long that I could see what the client meant. My report was full of technical detail, a brief executive summary that outlined the top vulnerabilities and that was pretty much it.

This started me a down a path of trying to analyze what a client is really trying to achieve when they have a penetration test performed, and consequentially how I can provide them the good value for their money.

Now is a good time to bring up an important point, the report you produce at the end of your penetration test is the only tangible artifact that you leave with the client. It doesn’t matter how good you were technically or how much you pwned them, if your report is garbage, you aren’t providing a valuable service, period. The report is one of, if not the most important parts of the penetration test.

So, let’s talk about some important considerations in producing a penetration test report that is provides some real value to your clients…

Know Your Audience

Understanding who will be digesting your penetration test report goes a long way in terms of knowing how to communicate your findings to your client. If the report is going to be consumed by someone at the director level and all you have in your report is technical vulnerability detail, they probably won’t read 90% of it, and worse off, they probably won’t see any real value.

But we’re trying to write a great report here, not just one targeted for a specific audience. With that in mind, I propose the following three audience types your report should address.

Technical Audience

This audience is pretty straight forward. They usually don’t care about pretty graphs or sexy templates. They aren’t afraid to roll their sleeves up and get their hands dirty, they want to know your findings in all of their technical glory and gruesome detail.

Managerial Audience

This audience is a little harder to nail down, as they can vary a bit, in general though, most of the managerial audience is interested in the ‘dashboard’ view of the results. They don’t really care about the technical details, they might peak at a few findings, but they are mainly interested in getting situational awareness.

Strategic Audience

This audience is focused on strategic initiatives and are usually in the executive / director seat. They generally don’t care about CVE-2099-1337 or how you managed to shell the thing with some other thing. This audience wants to know what strategic changes they can make in their organization to decrease vulnerability, and consequentially risk. They want to know about root cause problems, not vulnerabilities.

Now is another good time to stop and make something clear: vulnerabilities are not the real problem. It took me a while to realize. It wasn’t until I had been penetration testing for a few years before I finally understood this. If you perform penetration tests for the same client over and over again, you start to see a pattern. You find vulnerabilities, they fix them. You come back again, and find some new vulnerabilities, and they fix them. This process continues forever. Why you ask? Well, vulnerabilities are symptoms of another problem, but they are not the over-arching problem themselves.

As penetration testers, we have seen a lot of different networks, clients, and scenarios. After some time of being in this field you start to see things pretty clearly. For example, you perform a penetration test and find that over 100 hosts are critically vulnerable and many to different vulnerabilities. What does this tell you? it’s a pretty good indicator that the organization has a weak patch management program. Another example, you find that the systems are patched fairly well, but there are all sorts of security misconfigurations that allow you to abuse functionality and still pwn the organization. This is generally a good indicator that the client has a weak or non-existent configuration and build hardening standard.

Do you see what I’m getting at here? As a seasoned penetration tester, you can infer a lot of systematic issues that are the real problems leading to the vulnerabilities you find. These are the root causes. Fixing these issues is the only real way to make a lasting improvement in the client’s environment. I’m not saying that once root cause issues are fixed that you should never perform a penetration test again, certainly not. No matter how good your security program is, there is always room for improvement. That being said, if root cause issues are tackled instead of just focusing on the symptoms (read: vulnerabilities), you will find that overall the security posture improves and critical severity vulnerabilities a) exist for a shorter duration in the environment and b) are less common.

So let’s put two and two together. We’ve talked about target audiences, and about how we can infer some root cause problems. Should we include this stuff in our report? Absolutely, and not only should we bring up the root cause issues, but we should provide at least some high-level guidance on where they can begin to address them. By doing so, we can create a report that speaks to our client on different levels; from the technical to the strategic. And while we are at it, we can include some nice graphs and dashboards in the report to give our managerial audience the situational awareness they need, at a quick glance. If they want to dig deeper, they certainly can, but lets make sure we make the information accessible without having to spend 5 hours reading a report.

Okay, so our great report is going to speak to three different audience types, but how do we organize this information? I’m sure there are a number of ways, but I personally like to structure my reports in such a way that they get more technical the deeper you read in to them. Put the root cause (strategic) stuff up front, the dashboards and graphs (managerial) stuff right after, and then finally start jumping in to the technical rainbow land. This way our strategic audience can leaf through the first few pages, our managerial audience

Connect The Dots

Remember we talked about how you pwned the organization and hacked all the things? Great job… but hold on a second. Let’s say you found 100+ vulnerabilities, and you are a great gal / guy, so you go to painstaking lengths of explaining everything in detail to make sure that your client understands each and every vulnerability, and of course how to fix them! You’ve done a great job, right? I’d say you’ve done a good job, but there is still room for improvement, young jedi.

Think about it… 100+ vulnerabilities, some of which have write-ups 5+ pages long. How did you go from poking at their external web applications to pwning the entire domain? This is what I mean by connecting the dots. The report should walk the client through the penetration testing process, where did you start? what did you find? what did that lead you to?

I call this an attack narrative. I’m not saying I made up the term, it’s just what I call it. When we include this section in our great report, we are effectively narrating the penetration test to the client. This way they can understand our thought process and what we chained together to go from A to Z. Clients often find this detail very valuable as it allows them to understand the steps in a methodical manner.

Group The Things

Don’t just dump all the vulnerabilities in to one section. If you performed internal and external testing, looking at both the application and network layers; separate those findings! Your clients likely have multiple individuals who will be assigned to address vulnerabilities. So do them a favor and organize your findings in a logical manner, making it easier for your client to slice up the report and action it within their organization.

I hope you found some of this information useful. At the end of the day, we perform our penetration testing to make a difference, so make sure your reports and the services you provide help to make that difference.

I’ll follow up with another blog post shortly. There are some other points I’d like to bring up as well, so keep an eye out for part 2!

happy hacking