stories from the front lines: deploying an enterprise code - blog

YOUR

LOGO

HERE

Stories From the Front Lines: Deploying an Enterprise Code Scanning Program

Adam BixbyManager

Gotham Digital Science10/28/2010

Introduction

Adam Bixby, CISSP, MS

o Manager at Gotham Digital Science• Penetration Tester

• Static Analysis/Code Auditing

• abixby {at} gdssecurity.com

• http://www.gdssecurity.com

o Currently running an enterprise code scanning program at a large financial services organization

• Over 200 application integrated with code scanner in less than a year

Outline

Why am I here?

Lessons Learned

Tips for a successful scanning program Code Scanning Integration Strategy

Code Scanning Strategies - Pros and Cons

Questions

Why am I here?

Many companies try to invest in code scanning solutions only to fail

o Common Reasons for Failure:• Misguided perception of what a scanner can do

• Don’t know how to use it properly

• Don’t plan a pilot program

• Buy it without any training

• I’ll figure it out

• Don’t tune the product properly for their enterprise/application environment

• Fail to get support from upper management

What is not going to be discussed

Comparisons of the different Static Analysis Tools/Solutions

o Fortify SCA

o Rational AppScan Source Edition (Ounce)

o Vericode

o Checkmarx

o Coverity

o Klockwork

This talk is about the process

Lessons Learned

Lessons Learned

Need IT/Upper Management support

o Probably the most important takeaway from this presentation

o Without upper level support or mandate, code scanning programs are doomed to fail

• Development teams will not be as cooperative

• Remediation of issues need to be mandated

Issues will sit there scan after scan without being addressed

o All Integrations I’ve been involved in needed buyoff in order for their success

Lessons Learned

Common excuses from upper management

o Too expensive

o We are already performing penetration tests

o Application teams already have busy release schedules• Business units want functionality

• Security scanning will only slow them down

Lessons Learned

Common excuses from development teams

o We don’t have time

o Busy release schedule

o “We can’t fix all 1000 Cross-Site Scripting issues by our next major release”

o “This code has been deployed with these issues for 10 years, why do we have to fix it now?”

• Application makes us money, why change it?

Lessons Learned

Common assumptions that are made by team performing code scanner rollout

o All development teams are run the same

o Development teams will be onboard • Everyone wants to eliminate security issues, right?

o Our 5 year old spare server will be able to run a static analysis scan without any problems

• Code scanners are very resource intensive and hog A LOT of memory

o “Here’s a zip file with our code, this should scan properly”• Always missing libraries/dependencies

Tips for a Successful Scanning Program

Tips for a successful scanning program

Get management support ASAP

o Cannot stress this point enough

o Will make the difference between success and failure

o Who needs convincing?• CIO

• CISO

• Application Owners

• IT Management

o How do we go about convincing management?


Arm yourself with solid fact about code scanning solutions

o Help find and fix vulnerabilities

o Cost effective because detection and remediation comes earlier in the Software Development Lifecycle (SDLC)

• Fixes can be incorporated into regular bug remediation schedule instead of having to be performed out of band

Often the case when issues are found during pen tests

• If issue goes away in the code security scan, issue has been fixed. Not always necessary to allocated additional retesting time in release schedule

http://www.informit.com/articles/article.aspx?p=1357183

Study by Gary McGraw (Cigital) and Jim Routh, CISO for KPMG UShttp://www.informit.com/articles/article.aspx?p=1357183

Finding and fixing software security issues earlier in the SDLC makes economic sense!

“…the cost of removing a software defect grows exponentially for each downstream phase of the development lifecycle in which it remains undiscovered”

Actual cost benefit analysis at a large financial services firm based on real SDLC defect cost data

Cost to repair a security vulnerability in an application increases later in the development cycle (Forrester, 2009):

Want more evidence?


Arm yourself with solid facts about code scanning solutions, cont…

o Gets secure code to market faster

o Makes your applications and your company more secure from outside and inside threats

• This is why we are all here and why we have jobs…to keep bad guys out!

o Secure company data/secrets

o 75% of security breaches occur at the Application level (Gartner, 2005)



o Exploited security vulnerabilities in critical externally-facing applications can result in significant financial and reputation losses

o Regulators, Compliance, Audit, customers, partners, and security policies are demanding security solutions

• Audit will often inquire if source code scanning for security is part of a development lifecycle

• From an audit report I was privy to:

“…no requirements or guidance for regular security vulnerability assessments of source code and secure code development…”



o Vulnerabilities in code can be difficult for developers to identify without proper tools and training

o Penetration testing• Will not find all issues

• More expensive to rely solely on pen tests

Issues found in production are more expensive to fix

• Still should be performed since source code scanning will not find all issues (runtime bugs, server configuration bugs, business logic bugs, etc)



o “Our application team does not have time to setup and run a code scanning tool. Their schedule is full already.”

• Investing some minimal time upfront will save time in the future

• Scans can be totally automated

• Scan setup is usually not that complicated

Run in application build environment

• Process can be rolled into SDLC seamlessly



o Easy to enforce company’s existing coding practices and security policies without having to deviate from existing development lifecycle

• Finding security bugs can be done during normal build process

• Most code scanners allow for the creation of custom rules

create a rule to look for existing coding practices that need enforcing

• Fixing security bugs can be rolled into normal bug remediation effort without having to “stop the presses”

• Testing and verification can occur during testing and rescans


Learn how the tool works

o Sounds obvious, however, many security teams fail to get a code scanner integrated because they just are not proficient enough with it

• Don’t know how the scanner works and get frustrated

• Can’t troubleshoot scan failures

• Don’t know how to weed through the multitude of issues produced to identify most critical issues that need addressing


Educate Application Owners

o Armed with knowledge of the code scanner as well as facts about why you need a code scanner in your enterprise, inform app owners on what you are proposing

• Should get their buy-in as well

• Easy integration into development phase of SDLC

• Will find vulnerabilities during builds and help with remediation

• Facts stated previously

o Demo actual exploits of common vulnerabilities to show how dangerous they can be

• XSS and SQLi demos are easy and effective


Run a pilot program first

o Will help gauge how successful a full rollout across all applications will be

• Will determine if major roadblocks exist that will prevent full rollout

o Buy a few licenses upfront and purchase more in the future if needed


Run a pilot program first, cont…

o Identify a small number of highly visible, externally-facing applications within your organization and target them

• Get app owner or PM buy-in

• If these apps are successful, will give you good ammunition for convincing management to do a full rollout

• Try to find applications of varying languages

Want to test the code scanners ability against all of the types of applications run within your organization

a) C++, C

b) Java

c) C#, ASP.NET

d) PHP


Develop a step-by-step guide for application teams to utilize when they are walking through code scanner integrations

o This helps to make the integration process seamless and painless

o If you can do most of the work upfront for a developer, they will be much more receptive

Code Scanning Integration Strategy

Developed Web Site on company intranet portalo Identifies all steps needed from requesting code scanning integration

to issue remediation help

• Steps we have broken the integration process into:

1. Introduction to Enterprise Code Scanning program

• Includes description of scanner

2. Request code scanner integration within application

• E-mail that is sent to us

3. Instructions on how to install code scanner on Build Server/Desktop

4. Signup application developer(s) to attend training

• Identify member from each development team who is responsible for all code scanning issues (security guru)


Develop Web Site on company intranet portalo Identifies all steps needed from requesting code scanning integration

to issue remediation help

• Steps we have broken the integration process into, cont…:

5. Automate your code scans

• Automate the scans into existing build server

6. Scan code

7. Analyze scan results (validate issues)

8. Remediate issues

9. Submit feedback

10. Post Integration steps explained


Develop Web Site on company intranet portalo Identifies all steps needed from requesting code scanning integration

with the application to issue remediation help

• More on automating the code scans

Types of code scanning integrations:

• Continuous integration build server

• Build server using autosys/cron-job

• Desktop Scheduler on Windows machine

What happens to the scan results when scans are completed

• Automatically uploaded to a reporting management portal used for storing the results, generating reports and keeping track of key metrics

• Fortify 360 Server

• Rational AppScan Reporting Console


When utilizing build server automation, develop scan templates (batch files or shell scripts) for the developers ahead of time

o Allow them to download from company portal

o Makes for much faster integration

o Only a few details in scan script need to be replaced for scans to run successfully

o Minimize the potential for scan failure


Identify a member of your company’s security team to be the code scanner guruo Needs a strong technical understanding of security concepts

o Needs to be able to review the code scanner results and distinguish true positives from false positives

• Identify potential risk to the organization if issue is exploited

o Ability to provide technical guidance to application teams • Help understand and remediate security issues

o Ability to present results to application teams and provide remediation support

• Help prioritize issue remediation and provide recommendations


Develop a single scanning strategy and DO NOT allow for deviation

o Identify how your organization would like to run your code scans and force all development teams to use this method• Do not allow for one team to run their scans through build

integration and another team to run ad hoc scans using their IDE code scanner plug-in

o Lack of continuity will make troubleshooting scan failures harder

o Will produce inconsistent results


Developer runs code scans from their IDE using the integrated plug-ino Typically, developers pulls latest code from version control repository

into IDE and run scan

o Pros

• Easy to run

o Cons

• Usually different than what is in production

• Scanners are resource intensive

• Will hog up machine for long periods of time during the work day

• Developer needs to remember to run the scans and can only be done when the developer is using their workstation

• Cannot automate


Company Security Team runs scans for application teams from a dedicated machineo Pros

• Can build a powerful box that has a lot of hard drive space, memory and CPU power

o Cons

• Need to pull latest code drop from version control repository

Which version do you scan?

• Libraries/dependencies are not included in these code drops

The proper libraries will be needed every time you run the scan. Can be very cumbersome

• Too much overhead to keep all scans accurate and running


Run code scans on application’s existing build server and automate

o Recommended approach

o Pros• Source is the most up-to-date and accurate to what is in

production

• Code that is on the build server should be compile-able and therefore should generate the most accurate scans

All libraries/dependencies should be present


Run code scans on application’s existing build server and automate

o Pros cont…• Least time consuming out of all the approaches mentioned

Build server integrations are setup once

Scans can be automated to run weekly/monthly and automatically uploaded to your scan repository

• Most cost effective

Build servers tend to be beefier machines with high specs

No need to acquire an additional machines to run the scans

o Cons• None identified to date


Ensure scans are performed on a periodic basis

o Helps determine if remediation is occurring

o Does not have to been done during nightly builds

o Ensure new issues are not being added to code

o Tracks progress of remediation over time• Good for management level reports

o Ensure scans are actually being performed• Developers will run the scan once and forget about it

• Setup windows desktop scheduler, autosys, or use a continuous integration server

• Send weekly reminders to developer in charge of scanning if not using build server integration


Integrate security issues found by your code scanner into a bug tracking systemo Bugzilla, JIRA, etc

o Code scan issues become treated like any other bug found

• Hopefully given a higher priority

o Can be resolved and closed during normal bug remediation process

o Good place to track details of fix

• Can be shared across team/company

o Make sure you analyze bugs first before checking into bug tracking system

• False positive will make there way in there otherwise


Pay for Technical Supporto Will help iron out kinks/snafu’s that might arise

o Do not solely rely on the manual• Not written to give the user any more information than necessary to run against a simple

application

o May need custom rules to be written for unsupported libraries, policy enforcement, etc

o Management usually wants custom reports • Not the easiest to develop if you don’t know what you are doing

o Hiring a knowledgeable consultant to help with the integration goes a long way as well


Sensitivity/Classification of source code that is scanned needs to be identifiedo Need to treat the machine that code is scanned on as having the

same classification level as the source code

• Scanners leave translated versions of the source code on the scanning machine.

Files need to be treated with same classification as the source code

o Make sure the results file is given the same classification as well

• If code is SECRET, scan results need to be designated SECRET as well

• Code snippets, etc


Send application teams feedback surveys

o Feedback from the development teams can only help with making the code scanning program more successful

QUESTIONS?

Adam Bixby

abixby {at} gdssecurity.com

stories from the front lines: deploying an enterprise code - blog

Documents