By Caleb Sima

Falling Behind: The Challenge of Traditional PenetrationTesting in a Modern Software Development Environment

We need smart security researchers more than ever, but traditional testing delivery models simply don’t scale.

Penetration testing, or pentesting, has long been part of the application security mix. Though there may have been a time when we had hoped to reduce the need for manual testing through the use of automated scanners, in practice, the need for smart human testers has been proven over and over again. This is not because there is not value in application security scanning, it has its place in the application security mix, but rather that human creativity is required to find complex business logic issues that often lead to serious security flaws.

Pentesting is both tremendously valuable, and uniquely challenging. Ask a room of securityexecutives how one should find and deploy security researchers in the most effective way and you are bound to get a myriad of different responses. For many organizations, getting.

Pentesting has traditionally been delivered as a consulting service, though larger, more sophisticated organizations may have security researchers in house who can perform the same or similar tests, often referred to as a “red team.” For the purposesof this article, I will focus on outsourced, or external, pentesting. However, many ofthe challenges discussed can also apply to in-house testing, especially those related

Legacy Penetration Testing: No Pain, No Gain

“When we go through and prioritize the riskiest parts of the applications and the things that matter most, we do get pentesting coverage of those. But as you can imagine, there are tons of other things we still want to know about, but we just don’t have a way to get the coverage we need. We have got to figure out an approach for that and part of that is

just figuring out how to scale it - economically and with people.”

CHIEF SECURITY OFFICER

getting “smart human eyes” on their code is done through a complicated mix of outsourced penetration testing consultants, in-house teams, and more recently, bug bounty programs. And while many organizations have gotten better at finding flaws, they often lack a consistent, risk-oriented process for actually fixing issues once found and verifying they stay fixed.

Many security executives report that their efforts to find and fix security issues have only grown more strained as companies move toward a more continuous, iterative style of application development. Traditional pentesting requires extensive manual support for test administration and results mitigation. However, development rarely can pause for time-consuming security checks and testing activities that become too onerous for engineeringteams and are at risk of being bypassed to avoid production delays. It is like trying to stopa speeding car to check for tire damage on the fly, and knowing that even if you are luckyenough to slow it down enough for a good look, there may be a nail laying in wait less than a mile down the road.

In an effort to better understand this challenge and more specifically how companies are balancing the need for effective pentesting with the demands of the modern softwaredevelopment environment, I reached out to a number of highly respected security executivesto learn about how pentesting is managed in their organizations. What I found was that there is a gap between what companies want to do, need to do, and actually can do. For pentestingto adapt to the modern demands of software development, we must first work to understandthe areas where it falls short. This article will attempt to shed a little more light on currentchallenges.

to scalability. Pentesting has come to be seen as an essential part of the application securitymix, and is even mandated by a number of compliance standards, including the Payment Card Industry Data Security Standard (PCI DSS). Pentests are especially useful for providing depth of coverage for applications that require specialized knowledge to test effectively, such as mobile applications, because organizations can select researchers with specific credentials. Organizations also tend to have a fairly trusted relationship with the pentestingfirms, which typically vigorously vet testers and whose engagements are governed by contracts enforcing confidentiality. Because the security team can select researchers with specific talents and clearly define the scope of the testing, pentesting tends to have a higher “signal-to-noise” ratio than other forms of application security testing. But most importantly, there is simply no replacement for getting smart human testers using their own creativity to look for security issues in your code.

PENETRATION TEST VS RED TEAM

While these labels are sometimes

used interchangeably, red teaming and

pentesting typically have different objectives,

even if they rely on similar methods to

achieve them. Red teams are used to look for

a targeted, specific set of issues of concern

while pentesters are usually deployed with a

broader goal of finding any existing

exploitable vulnerability in the designated,

in-scope targets they are assigned to review.

Notably, they tend to have

different familiarity levels with the target(s)

they are testing and this can impact how the

findings should be interpreted in the

the context of risk assessment.

Yet, despite its obvious value, if you ask a room full of security leaders about their experiences with pentest-ing, be prepared to hear a bit of a groan. Managing a traditional pentest engagement strains resources andoften creates internal frustration as security and engineering teams work to balance their other responsibilities with the demands of supporting the test process and addressing its find-ings. First, there is a shortage of cybersecurity testing expertise, which means that highly qualified consulting firms are in high demand and often book projects weeks or even months in advance. Then once consultants are identified and contracted,onboarding the researchers and managing the test itself requires a high level of staff resources. Mostorganizations report that each step typically requires a highly manual process of emailing, conference calls, and spreadsheets.

It can also be a significant challenge to act on test findings. Consulting firms typicallyend their engagement with the delivery of test findings in a PDF document. Security and engineering teams are left to interpret the test results, prioritize findings based

on risk, fix issues, and retest and validate fixes. While some consulting firms deliver high-quality reports and provide some level of support during the mitigation process, security teamsfind that the quality of findings and level of researcher post-test support is highly variable. Many consulting firms focus on their ability to find security issues, and leave clients to handle findings interpretation, prioritization, and mitigation largely on their own.

While this has always been a challenge to manage for security teams, the move toward more iterative, continuous development cycles has only increased the strain reported. Given their resource limitations, most organizations must be strategic about how and whenpentesting is used, and are often unable to fully cover all of the code desired or review all of the changes they would like to have tested. Worse, the slow and sometimes inconsistent delivery of pentest findings threatens to erode its value further since “point-in-time” security checks lose relevance in a development environment focused on faster, more continuous delivery.

“I think better test integration and on-demand scanning would be a big helpin terms of efficiency.

I have 1.5 people who are really just PMing and dealing with the logistics of these tests - the

nuances of contracts, getting access, training findings, etc. So there is certainly more to be

said about moving moretoward self-service for the engineering teams and streamlining

workflow and integration.”

CHIEF SECURITY OFFICER

CHIEF SECURITY OFFICER

Bug Bounties to the Rescue?

“The economics of the traditional pentesting model just don’t work.”

Bug bounty programs have existed in some form since the nineties, and maybe even earlier,however their usage has grown considerably over the past decade and many now considerthem part of the mainstream application security mix. With a bug bounty program, anyone in the world can submit a potential security vulnerability to an organization, and if they are the first to find a valid bug, they will be paid a “bounty” and potentially recognized publicly for their work.

Bug bounties emerged out of a growing understanding that rather than fearing their

findings, companies could work collaboratively with the security researchers who were likely already poking around their products. Leading companies recognized that if they could create a more formal way of working with security researchers, they could not only improve the process of responsible vulnerability disclosure, but could also use their findingsto augment their application security programs. In fact, with an effective bug bounty program, you can fairly quickly attract a number of security researchers to look at your software and only pay them when a legitimate bug is found. In fact, some companies haveexplored whether bug bounties could potentially reduce their reliance on the more onerousprocess of pentesting.

In theory, bug bounty programs are a highly scalable method of application security testing. They provide a large number of human testers looking for a diverse set of issues across the full amount of code opened to the bounty. In addition, bug bounties are relativelyeasy to initiate and can be set up to run continually or quickly opened whenever a changedemands it.

However, security executives expressed a significant concern that test coverage will almost always remain unknown and unpredictable. This is because the provenance of the penetration testers and their tools will not be known, nor will the scope of issues and code they test against. As such, it is possible some sections of the application may remain untested or only scanned for a limited class of security flaws. For specialized applications,or sensitive internal applications,the inability to select or even vet researcher qualificationsmakes bug bounties particularly difficult to use. If a company wants someone who knows oauth to audit oauth - bounties don’t provide for this. Finally, bug bounties are not a practical solution for those wanting to perform security testing earlier in the developmentlifecycle because that requires providing a higher degree of access than organizations can’t comfortably grant to unvetted researchers.

While bug bounties are arguably easier to manage for internal teams than traditional pentesting engagements, they are not without their own difficulties. The first challenge is creating and maintaining program terms that are both fair and attractive and will draw researcher interest and retain it over time. The second challenge is planning for the unpredictable changes in the volume of effort required by internal staff to respond to findings and maintain productive communications with external researchers. Some organizations invest in a bug bounty platform to help outsource and streamline some aspects of program management.

Bug bounties also tend to have a low signal to noise ratio. An attractive program will produce a large number of findings, but they will need to be carefully reviewed and filtered for duplicates and false positives. Further, the delivery of test findings is unpredictable, straining the ability of organizations to manage the staffing and

“Do you have enough resources to actually run a penetration test against all of the assets that matter to you? I think that is the modern challenge of

penetration testing,”

CHIEF SECURITY OFFICER

resources needed to ensure timely remediation. So while bug bounties enable companies to quickly deploy smart human testers to find security issues, unpredictability in both thefrequency and quality of these findings often leads to a lengthy time-to-fix window for evencritical issues.

Given these challenges, security executives tend to view bug bounties as a complement to, not a replacement for, traditional pentesting. Both are valuable methods of applying humantesting at different points along the development lifecycle, and selecting one or the other should be based on specific testing requirements. For example, bug bounties are particularlyuseful in cases where speed of test execution is important, such as an initial review of newlyacquired software. However, any test that requires a high degree of internal access orspecialized knowledge, is likely better served by external pentesting with a trusted partner.

Every conversation I had with highly respected security executives confirmed that they want smart humans looking at as much of their code as possible, as often as possible. Butthey face significant constraints in achieving their goals. Put simply, traditional pentesting services are not cost effective or easy enough to use to scale. And while bug bounties do scale, you give up trust, depth of coverage, and expertise. And so, security executives are doing their best to test as much code as possible with an expensive and complicated mix of services while at the same time trying to manage the exposure they face from gaps in test coverage and frequency. And while many report that they are having increased success at finding flaws by deploying a mix of testing approaches, the positive security impact of testing remains limited by their inability to apply a consistent, risk-oriented remediation process. We can do better.

It is not the security value of pentesting that is in dispute - it is the delivery model. We need to combine the trust and known expertise we get with pentesting with the speed and scalability of bug bounty programs. Further, we need to apply a repeatable, consistent test workflow process that moves beyond just finding issues and encompasses the full mitigation process. We need Pentesting at Scale (PtaS).

Today’s Reality Falls Short, but Change is Coming

Pentesting at Scale is the ability to test all the code you care about, in a continuous manner,earlier in the development lifecycle. Imagine if your engineering team could click a buttonand request human eyes on a portion of code they just changed, knowing that they would quickly receive test findings and mitigation advice right on their screen, in tools they are already familiar with using. Security would be “cc’d” for awareness and feedback and significant issues would automatically be flagged for deeper security team review. Lengthytest scoping and administration processes are reduced to filling out a simple test requestform, and static PDF lists of findings are replaced by actionable change tickets with a buttonto click for rapid re-test and fix verification. Organizations would control the frequency of testing and cadence of remediation, enabling them to better manage staffing and resourcesto align with a more predictable and continuous find-to-fix workflow.

The ability to pentest at scale requires a modern delivery method for pentesting servicesthat provides on-demand testing in a self-service model. Fortunately, innovation is alreadyhappening in this area. Cobalt’s Pentesting as a Service is one example. With Cobalt, heavilyvetted domain experts are chosen from a carefully sourced pool of freelance researchers towork collaboratively on a penetration test of a selected application. This keeps costs low while still providing credentialed researchers and customer-defined, focused code coverage. Key elements of the test workflow - test planning, support and findings – are integrated and managed using a modern test delivery platform that integrates with the tools developers already use. Test findings are expert-reviewed and risk-prioritized before being reported and delivered as actionable tasks within the platform with support for developer-tester collaboration throughout the mitigation

BUG BOUNTYTRADITIONAL

PENTESTING

Slow

PTaSHighly Scalable

Quick Initiation and Results

Cost Effective

Unknown Provenance of Researchers

Low Signal to Noise

Unknown Test Coverage

Vetted, Qualified Researchers

Highly-Defined Test Coverage

High Signalto Noise

High Cost

Onerous TestingProcess

Scalable

Vetted, Qualified Researchers

Known Test

Coverage

Self-Service Model

Cost Effective

High Signal to Noise

Fast Results

process. However, integrating pentesting at scale into the application security mix will require more than just the right services and tools – it also demands a fundamental shift inhow we think about pentesting. We need to more effectively integrate security testing andits resulting remediation into the software development lifecycle so that large point-in-timesecurity checks are replaced with a more predictable and continuous find-and-fix workflowthat moves at the same speed as developers. A modern, proactive testing approach can’tmove forward if we stay stuck in a reactive legacy mindset.

Moving Forward

It is clear that traditional pentesting is facing a moment of needed change. Software development simply won’t slow down to accommodate a sluggish testingprocess, and overly onerous security activities are at risk of being bypassed or relegated to a once-a-year task that provides a check on a compliance audit, but little security value. Further, even as we continue to get better at finding security flaws, we must rememberthat test findings only provide security value if they lead to fixes that decrease risk. Simply finding issues more faster is only half the goal, we must also be able to address findings with the same speed.

To achieve this, the delivery model for pentesting needs to adapt to better align withmodern software development methodologies. Security teams should be actively working to support this innovation communicating their needs with their pentesting providers and working to identify opportunities to move pentesting earlier in their development processto unlock its strategic value. Customer-vendor collaboration aimed at modernizing pentesting must continue and is something I will work to encourage and support in future efforts. I look forward to continuing the conversation and sharing what I learn as I explore the concept of Pentesting at Scale further.

Caleb Sima has been engaged in the Internet security arena since 1994 and has become widely recognized as a leading expert in web security, penetration testing, and the identification of emerging security threats.

His pioneering efforts and expertise have helped define the web application security industry.

Caleb Sima's latest role was at Capital One serving as the Managing Vice President of Cyber Security. Prior to Capital One, he was the CEO & co-founder at Bluebox Security (acquired by Lookout) and previously operated as CEO of Armorize (acquired by Proofpoint). In the past, he served as CTO of Application Security at Hewlett-Packard via the acquisition of his first startup where he was CTO & Founder of SPI Dynamics.

As a founder of Badkode Ventures, Caleb also invests in startups. Some notable investments are FOSSA, Pindrop Security, Purewire, and Rocana.

Visit Cobalt.io to learn more about how Pentest as a Service helps scale security testing.

About the Author