Category Archives: Security Testing

Pen Testing is Dying- Here are the Six Things that are killing It

Don’t all kill the messenger at once but the sad truth is there are a growing number of organizations that are not seeing the ROI in pen testing and I’m afraid they have good reasons.

There is a concept in organizational behavior about there being two reasons for a culture’s behavior- the reason they give you when asked and the answer that is not verbalized.

When leaders are asked why they are decreasing their spending on pen testing the answers I am hearing sound very reasonable- the attack surface is too broad, the number of attack vectors has grown to an untestable number, etc etc. No argument here.

However, I would argue that there are also some other, probably bigger reasons that have a larger influence that do not get verbalized. In fact, I think there are six of them.

  1. Supply and demand created a shortage of skilled testers

The security boom we’ve all been enjoying has created a huge need for security professionals.  As a security professional I’m loving it. However, like all skills shortages there is a vacuum moving the bar for “senior level” down lower and lower. What was once considered an intermediate level tester is now considered a senior person.

This is a reality that has no good answer. Personally, I prefer to have testers who have their OSCP certification but with supply and demand this has become a luxury, not a baseline.

  1. Confusing automated assessments with pen testing

I find that the actual definition of what a pen test is has been evolving and some of that evolution seems to parallel the supply and demand issue. What we used to refer to as automated assessment tools have now become the pen test. Scanning a target with nmap, Nessus and Burp and pasting the results in to a template is not a pen test. There is certainly value in these activities as one of many parts of a holistic assessment but calling it a pen test has diluted the value of the term.

  1. The Rules of Engagement for pen tests typically protect the weakest link

Pop quiz- name a recent major breach that didn’t involve a targeted email as the point of entry? Yea, me neither.

However, most pen test will have rules of engagement prohibiting spear phishing employees and the use of backdoors. I strongly suspect the perception of a pen test report would jump in value if it contained screenshots from the meterpreter session running on the CFO’s laptop while he transfers funds between banks. There are some really good reasons pen tests can’t mimic real life attacks but because of this their value is reduced.

  1. Length of engagements do not match real life

Another aspect of pen testing that doesn’t match real life threats are the length of engagement. In real life attacks, the bad actors don’t have a time limit. They can spend months or even years on their target. In a pen test, the tester has a week or two to do his assessment and write up the findings. We can’t expect a team of testers to be able to cover all possible vectors in a few weeks.

  1. Lack of incentive.

Billable hours pay your bills but they don’t exactly light a fire under an analyst’s rear end. I’ve often wondered what an assessment would look like if the client set up a pen test like a bug bounty, providing the team incentives for what they found.  When a team of bad actors operating out of Eastern Europe can retire after one big score and you have pen testers working for billable hours there is a motivation mismatch.

  1. Pen Testing to Check a Box for your Auditor

For a pen tester to do a great job they need to be able to follow where their findings and instincts take them. A “pen test by numbers” will appease a requirement but unless it’s done with passion and purpose it’s not going to be great.

 Do I think pen testing is still valuable? Absolutely. But for all of the reasons above I think it is getting a bad rap. Disagree with me? I’d welcome your thoughts. @CyberSecOlogy #pentesting. Contact me via the form below:
 

 

Google’s Firing Range Test Site

[ Giigle Firing Range: Download it | Hack it | Alternatives to it]

Google's Firing Range

Yesterday, Google Security Engineer Claudio Criscione released version 0.42 of the web scanner test application called Firing Range. The author of the test app describes Firing Range as “a Java application built on Google App Engine and contains a wide range of XSS…” They make the source available on github or there is a public website available for target practice.

At first glance, Firing Range looks sparse. However, those who don’t look past the landing page and launch a tool or two are not going to be able to appreciate the elegance of the test site. Firing Range is a unique and valuable addition to the web’s portfolio of test sites.

While the Hackazon test site provides what I have described as a “torture test” for scanners Firing Range is more of a “Rubik’s cube.” Most scanners will complete a test of Firing Range in well under an hour and nothing about the site is resource intensive or attempts to hide vulnerabilities from the test tool. Instead, Firing Range takes the opposite approach and focuses on completeness of the assessment. That is, every test page is directly available within the first two levels of the landing page and all the test pages are labeled for the target test. For example, all of the pages that test for reflective cross site scripting are available from the reflected cross site scripting page. This page is then broken in to clearly defined sections:

Firing Range Screen Capture

Firing Range on its own should not be considered a single source for testing a scanner (and I do not believe the author ever meant for it to be). However I believe that it will soon become a gold standard when assessing a scanner’s ability to detect and report on cross-site scripting.

Google's Firing Range

Pros:

  1. For evaluating a scanner’s strengths and weaknesses in detecting and reporting on cross-site scripting, you won’t find a better tool than Firing Range.
  2. The author has carefully and thoroughly labeled each test and offered explanations where needed.
  3. Firing Range allows an assessor to map the true positive findings and false negative findings.

 

Cons:

  1. Firing Range makes no attempt to test a scanner’s ability to test in real life situations such as complex workflows like you would find in an eCommerce site.
  2. Firing Range tests very little beyond cross site scripting. It would be great to see this model extended to other vulnerability families.