Dynamic Security Scanning in a CI: ZAP Scanning with Jenkins.
Today, I will walk through configuring a daily DAST scan against an application, using Jenkins and ZAP. The process can be used similarly with any DAST scanner, depending on how the specific scanner is setup.
This is the second part of a series. In part one, I walked through initial Jenkins setup, and creating a build pipeline that incorporates two static code scanning tools for security.
Although it is possible to include a dynamic scan directly within the build pipeline, I prefer to make it a separate job that runs daily. This is because I prefer to run in depth scans, which means they could take a while.
Dynamic scanning and why you should automate it
Dynamic scanning is essentially hitting an app with a battering ram and seeing what happens. DAST scanners run a series of common attack strings in various input forms, header data, and GET requests and sees what comes back.
This can show vulnerabilities that were missed in static scans, or where input/output filtering is missing coverage that might allow a bypass.
Dynamic scanning, however, can take some time to run, and by it's nature impacts and potentially damages an application (because it submits forms). As a result, it should always be run against a non-production instance that will not do annoying things like send emails to users on form submit.
It also typically needs one or more user accounts to test properly. Ideally, one user per application role should be configured to ensure full coverage.
Automating this process allows someone who is knowledgeable about the scanner configuration to work with someone knowledgeable about the application to create a finely tuned scan. Once created, it can be run repeatedly and the benefits distributed to all team members.
Setting up the environment
You can find all the configuration files and code for this post in my github, if you want to follow along and configure this on your own machine.
For this exercise, we will need a few services running:
- Jenkins as our CI/CD tool
- A ZAP service for dynamic scanning (we will setup a separate ZAP node in this post, though alternately a Jenkins slave could be configured with ZAP.)
- WebGoat / WebWolf as our vulnerable application
- SonarQube, which we won't use here. It was used in the last post and remains here so all pipelines can continue to run successfully.
I configured all of these in a docker-compose file below. This ends up as five containers, which may strain some machines. You can safely remove SonarQube, and possibly WebWolf if you wish to reduce system load.
version: '3'
services:
jenkins:
build:
dockerfile: jenkins-dockerfile
context: .
ports:
- 8080:8080
- 50000:50000
volumes:
- jenkins_home:/var/jenkins_home
sonarqube:
image: sonarqube:7.7-community
ports:
- 9000:9000
zap:
image: owasp/zap2docker-weekly
ports:
- 8000:8000
# We start a ZAP daemon that can be connected to from other hosts. We will connect to this from Jenkins to run our scans
entrypoint: zap-x.sh -daemon -host 0.0.0.0 -port 8000 -config api.addrs.addr.name=.* -config api.addrs.addr.regex=true -config api.key=5364864132243598723485
volumes:
- zap:/zap/data
# docker-compose information taken from the project at https://github.com/WebGoat/WebGoat
webgoat:
image: webgoat/webgoat-8.0
environment:
- WEBWOLF_HOST=webwolf
- WEBWOLF_PORT=9090
ports:
- "8081:8080" # Port changed from 8080 on localhost so as not to conflict with Jenkins
- "9001:9001"
volumes:
- webgoat:/home/webgoat/.webgoat
webwolf:
image: webgoat/webwolf
ports:
- "9090:9090"
command: --spring.datasource.url=jdbc:hsqldb:hsql://webgoat:9001/webgoat --server.address=0.0.0.0
volumes:
jenkins_home:
webgoat:
zap:
These are mostly standard images. I modified the Jenkins one with a custom dockerfile to include python and the ZAP-CLI tool. In a production instance, we could manually install this on our deployed Jenkins, create a dedicated ZAP Jenkins slave, or use this dockerfile if doing a dockerized deployment.
FROM jenkins/jenkins:lts
USER root
RUN apt-get update
RUN apt-get install -y python-pip
RUN pip install --upgrade pip
RUN pip install --upgrade zapcli
We can now launch our services:
$ docker-compose build
$ docker-compose up
Assuming everything works, you can test a few URL's:
- Jenkins: http://localhost:8080
- WebGoat: http://localhost:8081
- ZAP Proxy: http://localhost:8000
Configuring and running ZAP scans through docker
Before we configure anything in Jenkins, let's play around with the ZAP docker image and make sure we can run scans successfully against WebGoat. We can then operationalize that into regular Jenkins scans.
If you inspect the docker-compose file above, in the zap image definition, the entrypoint specifies an API key, which you should change and randomize for your own tests. The port to bind to is also specified, along with a configuration option that tells ZAP to allow connections from outside the container.
WebGoat requires a user be logged in to view most information. For ZAP to successfuly scan those pages, it requires importing a context file that includes information like
- login page
- regular expressions that tell ZAP whether we are logged in or logged out
- working login credentials
Create a user on the WebGoat instance with credentials "tester / tester", which is what I have used in the sample context file. If you want to generate your own context file, the ZAP documentation has a good walkthrough for creating it. For WebGoat, I created the following context file, and named it WebGoat.context:
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<configuration>
<context>
<name>WebGoat</name>
<desc/>
<inscope>true</inscope>
<incregexes>http://webgoat:8080.*</incregexes>
<tech>
<include>Db</include>
<include>Db.Firebird</include>
<include>Db.HypersonicSQL</include>
<include>Db.IBM DB2</include>
<include>Db.Microsoft Access</include>
<include>Db.Microsoft SQL Server</include>
<include>Db.MySQL</include>
<include>Db.Oracle</include>
<include>Db.PostgreSQL</include>
<include>Db.SAP MaxDB</include>
<include>Db.SQLite</include>
<include>Db.Sybase</include>
<include>Language</include>
<include>Language.ASP</include>
<include>Language.C</include>
<include>Language.PHP</include>
<include>Language.XML</include>
<include>OS</include>
<include>OS.Linux</include>
<include>OS.MacOS</include>
<include>OS.Windows</include>
<include>SCM</include>
<include>SCM.Git</include>
<include>SCM.SVN</include>
<include>WS</include>
<include>WS.Apache</include>
<include>WS.IIS</include>
<include>WS.Tomcat</include>
</tech>
<urlparser>
<class>org.zaproxy.zap.model.StandardParameterParser</class>
<config>{"kvps":"&","kvs":"=","struct":[]}</config>
</urlparser>
<postparser>
<class>org.zaproxy.zap.model.StandardParameterParser</class>
<config>{"kvps":"&","kvs":"=","struct":[]}</config>
</postparser>
<authentication>
<type>2</type>
<loggedin><a role="menuitem" tabindex="-1" href="/WebGoat/logout">Logout</a></loggedin>
<loggedout><button class="btn btn-primary btn-block" type="submit">Sign in</button></loggedout>
<form>
<loginurl>http://webgoat:8080/WebGoat/login</loginurl>
<loginbody>username=tester&password={%password%}</loginbody>
</form>
</authentication>
<users>
<user>2;true;dGVzdGVy;2;dGVzdGVy~dGVzdGVy~</user>
</users>
<forceduser>2</forceduser>
<session>
<type>0</type>
</session>
<authorization>
<type>0</type>
<basic>
<header/>
<body/>
<logic>AND</logic>
<code>-1</code>
</basic>
</authorization>
</context>
</configuration>
If you inspect this file, you will see the docker-compose network address for the site, login and logout regex patterns, a user account, and login form address. Most of this is created by the ZAP GUI automatically, though I had to change the hostname manually from localhost, since we will install this on the ZAP host.
We next want to share this file with the ZAP container, so we have to put this context file in the shared volume. Find the volume location, and copy the xml file to it:
$ docker inspect dast_pipeline_example_zap_1 | grep data
"dast_pipeline_example_zap:/zap/data:rw"
"Source": "/var/lib/docker/volumes/dast_pipeline_example_zap/_data",
"Destination": "/zap/data",
"/zap/data": {}
$ sudo cp WebGoat.context /var/lib/docker/volumes/dast_pipeline_example_zap/_data
$ docker exec dast_pipeline_example_zap_1 ls data
WebGoat.context
Now we have to import the context to the ZAP process. This only has to be done once (though it will have to be re-done if you restart the container).
$ docker exec dast_pipeline_example_zap_1 zap-cli --api-key 5364864132243598723485 --port 8000 context import /zap/data/WebGoat.context
[INFO] Imported context from /zap/data/WebGoat.context
We can now run a scan from within the docker container. This will ensure that the ZAP container can successfully reach the WebGoat container and that scanning works correctly.
$ docker exec dast_pipeline_example_zap_1 zap-cli --api-key 5364864132243598723485 --port 8000 quick-scan -c WebGoat -u tester -s all --spider -r http://webgoat:8080/WebGoat
[INFO] Running a quick scan for http://webgoat:8080/WebGoat
[INFO] Issues found: 8
+---------------------------------------+--------+----------+---------------------------------------------------------+
| Alert | Risk | CWE ID | URL |
+=======================================+========+==========+=========================================================+
| SQL Injection | High | 89 | http://webgoat:8080/WebGoat/login?logout=+AND+1%3D1+--+ |
+---------------------------------------+
.... (output truncated)
Now that we know the scanner is working, we can run zap-cli locally and connect to the container, to ensure that the ZAP process accepts connections from the network. You can skip this step if you don't want to install zap-cli on your machine.
$ zap-cli --zap-url localhost -p 8000 --api-key 5364864132243598723485 quick-scan -c WebGoat -u tester -s all --spider -r http://webgoat:8080/WebGoat
[INFO] Running a quick scan for http://webgoat:8080/WebGoat
[INFO] Issues found: 18
... (output truncated)
Setting up a Jenkins Pipeline with ZAP
Now that we have a ZAP container scanning WebGoat successfully, let's automate this and put it into a pipeline that runs daily.
Log into the Jenkins container (http://localhost:8080) and create a new item of type Pipeline.
I recommend running the DAST scan daily and storing the results in an HTML report on Jenkins that the whole team can view. To setup daily runs, in the pipeline configuration set build triggers -> build periodically, with the following schedule:
H H * * *
This let's Jenkins determine the starting hour/minute for the scan based on overall system load, but will always run the pipeline once a day.
In order to save the HTML output easily, I installed a new plugin, HTML Publisher. With this plugin installed, paste the following into your pipeline script textbox:
node {
script {
try {
// Context import fails if it already exists
sh 'zap-cli --zap-url zap -p 8000 --api-key 5364864132243598723485 --port 8000 context import /zap/data/WebGoat.context'
}
catch (Exception e) {
}
}
script {
try {
// If it finds results, returns error code, but we still want to publish the report
sh 'zap-cli --zap-url zap -p 8000 --api-key 5364864132243598723485 quick-scan -c WebGoat -u tester -s all --spider -r http://webgoat:8080/WebGoat'
}
catch (Exception e) {
}
}
sh 'zap-cli --zap-url zap -p 8000 --api-key 5364864132243598723485 report -o zap_report.html -f html'
publishHTML([
allowMissing: false,
alwaysLinkToLastBuild: false,
keepAll: false,
reportDir: '',
reportFiles: 'zap_report.html',
reportName: 'ZAP DAST Report',
reportTitles: ''
])
}
Note that all commands are using the docker-compose network configration names - based on service names in the docker-compose file. Thus "zap" resolves to the ZAP container, and "webgoat" to the WebGoat container. Change these to match your actual environment if you modified the compose file.
We also wrap our ZAP commands in try/catch blocks - the context import is needed in case the service is restarted, but in other cases it would fail and stop the build. The scanner command returns an exit code 1 (error) when it finds issues, which would stop the build. Since we always want to generate and save the HTML report afterwards, so we also catch and ignore this error.
You can run the build immediately, and should see a new link when the run completes - ZAP DAST Report. Clicking that gives the HTML vulnerability report that was generated.
Conclusions
Unfortunately, I don't think the scan results are all that great for WebGoat. Not many things were found considering this is a known bad application with many kinds of vulnerabilities that are created for learning. Regardless, I always like to have some kind of scanner running, where results are reviewed regularly.
Although ZAP didn't find as much as I would have liked, the overall process of integrating scanners and scheduling runs is highly valuable. Most DAST tools I have worked with have API tools similar to ZAP that allow remotely kicking off a job and returning HTML results.
Now we just have to create a culture of regularly reviewing the results on the project team!