Successfully Starting a Secure DevOps / DevSecOps Program

When I was first asked to build a Secure DevOps program, I was excited about the possibilities for continuous security, and truly making an impact on our applications from the first line of code written.

Over the following months, the difficult reality set in for me: Though there is no shortage of upbeat vendor presentations and articles, actual implementation is messy.

In this article, I will talk about some approaches I tried that failed, and some approaches that have worked well in a variety of environments across a wide range of methodologies.

Get to know your engineering teams

The first thing I went about doing was directly embedding myself and my team members into existing development life-cycles in a variety of teams. I wanted a deep understanding of their processes, tools, and challenges, which we could only achieve by actually contributing code and following their release process.

We started with a team building power plant software, a team building a complex micro-service based web application, a team customizing vendor software, and a team with a non-technical PM and 100% outsourced technical team.

We were hoping to find a lot more commonalities than we did - each team was using different release processes, source control products, deployment platforms, and had wildly different technical maturity.

This was extremely valuable for understanding what solutions would be viable for our broader organization, and immediately eliminated certain solutions from our plans.

Deeply understand why CI/CD exists

Of the four teams we worked with, one was using a Jenkins they maintained themselves, one was using a Jenkins instance maintained by a shared DevOps team, one was using Foundation Server, and one was using custom scripts.

My first thought was that if we could standardize CI/CD offerings across the business, and automatically include the use of security tooling in that central solution, then life would be good.

As we worked within the release processes directly, however, it became increasingly clear that the nature of the tooling and processes was impossible to both standardize and maintain the core purpose of the tools: to enable high quality, fast releases in their specific operating environment, which varies wildly between teams.

Once it became clear that, at least for most of our teams, a central security solution would be too significant an undertaking for too little potential reward, I instead started thinking of how to scale a process that would work for all kinds of teams without a central authority.

Tackle challenges with manual effort first

We started seeing true success when we put aside our desire to automate everything first and just started using tools manually with each team.

Because teams had a wide variety of technologies, this allowed us to have honest conversations about security tool value with each team and understand their unique needs.

What emerged from that effort was a better understanding of the best uses cases for each tool.

We started with Coverity for SAST analysis, WhiteSource for OSS, and TinFoil for DAST, along with a number of language and framework specific tools. These are expensive (and good!) solutions, so they may be out of reach for some teams. I maintain a list of DevOps security tools that are free and open source for that reason.

Automate to maximize tool use

What we discovered was that, unfortunately, too many of our teams were not planning to use a full CI/CD pipeline in the near future, which meant that there was no easy way for us to easily automate the use of our security tools.

Instead, we wrote a custom software connector that would hook into our enterprise github environment and trigger security scans on code change, then collect that data in a central repository per application across all security tools we supported.

This allowed us to use common security tools across teams that were using CI/CD and those that weren't, with similar levels of security tool usage. Because we had spent time working with these same teams, they were excited to help us and work with us.

Be the first to implement fixes

Whenever feasible, have the security team attempt to implement fixes. Sometimes this will be a pull request to the engineering team. The security team member should go through all the normal code reviews and release processes that other team members would go through - and use this as an opportunity to teach the engineering team why you chose the solutions you chose.

I know this takes a lot of effort and isn't scaleable, but it will build rapport and expertise with the right teams. If you do this right, over time there will be less and less work to do.

Focus on culture change

The engineering team is your first line of defense. Security professionals have to be careful about the kinds of comments we make about other people's work. I have been in too many conversations where security folks have disparaged engineering folks.

I have yet to meet a developer who wanted to create an insecure app. Traditional development teams often just do not focus on security issues during education, code review, or performance evaluation.

Senior individuals may not know more than their junior counterparts, and may be embarrassed finding that they created an "obvious" security hole.

But engineers love to be the best they can be in their craft, and appreciate being shown how to improve as long as it is not done in a condescending way. If you present the data in a neutral way, and offer them an honest path forward, it will be appreciated.

One tool I have found useful is transparency. Once you have vulnerability data, show it to the team that produced it, and work with them to fix it. Nothing beats working shoulder to shoulder to create lasting change.

That should give you a strong start on a long journey!

The end goals of a DevSecOps program is fully automated security instrumentation at every level of development - from the moment the developer opens an IDE to the moment the deployed infrastructure is shut down and everything in between.

There are many paths there, but I have found that trying to start small, with manual efforts that don't scale build the fundamental relationships and understanding that is often missing.

Once you have those in place, keeping up a conversation and building more advanced pipelines and tooling becomes a continuous uphill climb together.

What has worked for you? I am always looking for better approaches, so reach out and let's talk!