Deploying Docker Securely
Docker is a fantastic piece of technology for teams. Some time ago, I published a series of articles on building a docker security program, which covers doing a threat assessment, image static and runtime analysis, and overall container patching and maintenance.
I was asked recently about deployment security, which was missing from the original series. In this post, I will outline steps you can take to securely deploy a docker environment. This assumes you are using Docker daemon on hardware running Linux that you administer. Our main goal is to prevent container escapes and multiple container breaches in the event that a single container is compromised.
Maintain the Host & Daemon Software
It should go without saying, but protecting the host is as important as always. Docker regularly releases security patches for the daemon, so ensure you have a good strategy for updating and maintaining the environment.
- Regularly patch the docker software installed on the server.
- Regularly update the underlying operating system and software
- Harden your server and setup host based intrusion detection, with docker configurations.
Don't Run in Privileged Mode
Privileged mode gives a container access to many capabilities reserved for root on the host machine, such as control of all devices, the ability to create and manage other containers, and exceptions to any resource limitations imposed by cgroups.
Device control can be used to do all sorts of unfriendly things, such as change the host partition table, mount and read or modify sensitive host file systems, etc.
If at all possible, don't run containers in privileged mode. Some applications may require this and have a valid use case, such as fully managing other docker containers. If so, don't place containers unrelated to the privileged one on the same host, to limit the damage if the privileged container is breached.
In general, it is better to selectively add capabilities to the container, rather than giving blanket privileged access. You can see all default and addable privileges in the runtime documentation.
Remove Default Capabilities. Add back what is needed
In the same vein as above (adding capabilities as needed instead of running privileged), I recommend removing all default capabilities and only adding what will be used - frequently no capabilities will be needed in production.
Docker containers run with the following capabilities by default:
| Capability | Description |
|---|---|
| SETPCAP | Allow a process to change it's own capabilities set (within the set it is already allowed). Should not be dangerous in practice. |
| MKNOD | Allows creation of special devices. Almost all containers will receive the devices they need automatically, so this is almost never needed. |
| AUDIT_WRITE | Allow the container to write records to the underlying kernel auditing log. This is generally used for things like ssh login. It is a better practice to keep all container logs within the container and drain them, rather than allow the container to write to the host log files. |
| CHOWN | Allow the container process to change file permissions. This should generally never be needed in production. |
| NET_RAW | Allows RAW and PACKET sockets, which can be used to spy on networks or create nasty packets. Some applications may need this, but most won't. |
| DAC_OVERRIDE | Allows the root user to bypass normal file permission checks. No container should ever need this. |
| FOWNER | Bypass permission checks on operations that normally require the UID of the process to match the UID of the file. |
| FSETID | Don’t clear set-user-ID and set-group-ID permission bits when a file is modified. This should only be needed by installations, so generally not in production. Allowing this can let a user modify privileged programs. |
| KILL | Allow the root user to send kill signals to non-root processes. Not generally needed if keeping one container per process, but not too dangerous in general. |
| SETUID | Allows running of setuid binaries, which allows a process to change it's effective user id. This is required for some processes that require privileged ports (less than 1024) and by some processes like Apache to start. It should only be given to containers that need it. |
| SETGID | Just like setuid above, but for group ids. |
| NET_BIND_SERVICE | Bind to ports less than 1024. Enable this if your container runs a service that listens in the this port range only. |
| SYS_CHROOT | Allows use of the chroot command. Enable only if your process actively uses chroot to change the root directory. |
| SETFCAP | Allows the container to set file capabilities. Sometimes this is needed during a software install, but generally should not be needed by a running container in production. |
As mentioned, it's best to drop all of these and only selectively add the specific capabilities needed. This takes a little more management overhead, because you may have to work with the application team to determine what is really needed.
Run a container with no capabilities:
# docker run --cap-drop=all my_container
Add back in only net_bind_service and setuid capabilities:
# docker run --cap-drop=all --cap-add net_bind_service --add-cap setuid my_container
Mount Storage Carefully
Some Docker guides will walk you through how to mount storage from various locations. This is generally fine when done correctly, but can lead to potential issues. A few things to be wary of:
- Don't mount the Docker socket file - any process with read/write access to the socket file also has admin rights over docker processes on the same host. This may be OK if the container mounting the socket is trusted and it's purpose is to manage other containers on the same host.
- Don't mount sensitive host file systems - It's best to mount only a clean filesystem to the container, that is not shared with the host for any reason except being container storage.
- Don't mount file systems to more than one container - It may be tempting to mount filesystems to multiple containers to share files, but do so with care, as files modified in one will also be modified in the other.
Only Run Trusted Images
We want to avoid a scenario where an attacker could modify images hosted for deployment, or convince the host to startup a container an attacker specified from an untrusted source. The solution to this is Docker Trusted Images and image signing.
Setting up this process takes a little work - we will need to create offline master keys along with a key creation and rotation process for engineers who will be creating and uploading images, so it's not for every organization.
The Docker documentation has an excellent walkthrough for setting up a trusted registry and the various keys and infrastructure required to do this. Beause of the added risk of losing a master key, this is something you should embark on with a good understanding of chain of trust and key management in your organization.
Be Aware of Docker Networks
Tools such as docker compose can help glue containers together by creating docker specific networks. These networks can provide decent network segmentation out of the box when networks are created with care, but create a single large network by default.
Because the default configuration is one network for all containers, it is easy to create a container network that allows communication between any containers on any port, which is not monitored by traditional network tools. This is especially true with containers running on the same host, where no actual network traffic is generated, but container separation is depended upon.
To mitigate this, leverage orchestration tools that offer networking definition tools, or work with teams to ensure that networks are being well defined between various containers.
Final Words
I recommend you also check out the official docker security documentation. It has a wealth of information and can expand on these topics including more information about the underlying mechanisms.
If you find a conflict between the documentation and anything I have said, the documentation wins (but please comment here and let me know!).
Please comment with other best practices I may have missed or glossed over!