Don’t all kill the messenger at once but the sad truth is there are a growing number of organizations that are not seeing the ROI in pen testing and I’m afraid they have good reasons.
There is a concept in organizational behavior about there being two reasons for a culture’s behavior- the reason they give you when asked and the answer that is not verbalized.
When leaders are asked why they are decreasing their spending on pen testing the answers I am hearing sound very reasonable- the attack surface is too broad, the number of attack vectors has grown to an untestable number, etc etc. No argument here.
However, I would argue that there are also some other, probably bigger reasons that have a larger influence that do not get verbalized. In fact, I think there are six of them.
- Supply and demand created a shortage of skilled testers
The security boom we’ve all been enjoying has created a huge need for security professionals. As a security professional I’m loving it. However, like all skills shortages there is a vacuum moving the bar for “senior level” down lower and lower. What was once considered an intermediate level tester is now considered a senior person.
This is a reality that has no good answer. Personally, I prefer to have testers who have their OSCP certification but with supply and demand this has become a luxury, not a baseline.
- Confusing automated assessments with pen testing
I find that the actual definition of what a pen test is has been evolving and some of that evolution seems to parallel the supply and demand issue. What we used to refer to as automated assessment tools have now become the pen test. Scanning a target with nmap, Nessus and Burp and pasting the results in to a template is not a pen test. There is certainly value in these activities as one of many parts of a holistic assessment but calling it a pen test has diluted the value of the term.
- The Rules of Engagement for pen tests typically protect the weakest link
Pop quiz- name a recent major breach that didn’t involve a targeted email as the point of entry? Yea, me neither.
However, most pen test will have rules of engagement prohibiting spear phishing employees and the use of backdoors. I strongly suspect the perception of a pen test report would jump in value if it contained screenshots from the meterpreter session running on the CFO’s laptop while he transfers funds between banks. There are some really good reasons pen tests can’t mimic real life attacks but because of this their value is reduced.
- Length of engagements do not match real life
Another aspect of pen testing that doesn’t match real life threats are the length of engagement. In real life attacks, the bad actors don’t have a time limit. They can spend months or even years on their target. In a pen test, the tester has a week or two to do his assessment and write up the findings. We can’t expect a team of testers to be able to cover all possible vectors in a few weeks.
- Lack of incentive.
Billable hours pay your bills but they don’t exactly light a fire under an analyst’s rear end. I’ve often wondered what an assessment would look like if the client set up a pen test like a bug bounty, providing the team incentives for what they found. When a team of bad actors operating out of Eastern Europe can retire after one big score and you have pen testers working for billable hours there is a motivation mismatch.
- Pen Testing to Check a Box for your Auditor
For a pen tester to do a great job they need to be able to follow where their findings and instincts take them. A “pen test by numbers” will appease a requirement but unless it’s done with passion and purpose it’s not going to be great.
Do I think pen testing is still valuable? Absolutely. But for all of the reasons above I think it is getting a bad rap. Disagree with me? I’d welcome your thoughts. @CyberSecOlogy #pentesting. Contact me via the form below: