holes in container images. The third major aspect of container security is at least as important from an operational perspective. That is the need to capture and potentially automatically remediate any issues when anomalous behavior is discovered from your running containers.
Only a handful of trustworthy and battle-worn container runtime security applications exist. Of those there is one Open Source tool that stands out from the crowd. Created by a company called Sysdig (sysdig.com) in 2016 and a member of the Cloud Native Computing Forum (CNCF), Falco (falco.org) excels at both container and host security rules enforcement and alerting. Of the more popular commercial tools there are Prisma Cloud Compute Edition (formerly Twistlock prior to acquisition) and Aqua from AquaSec.
Falco (sysdig.com/opensource/falco) offers exceptional Open Source functionality that can be used to create rulesets to force containers to behave in precisely the way you want. It also integrates with Kubernetes API Audit Events, which means that all sorts of orchestrator actions can be secured in addition. You can find more information here:
falco.org/docs/event-sources/kubernetes-audit.
In this chapter, we will look at installing Falco and then explore its features and how it can help secure our container runtime and underlying hosts, in the same way that some commercial products do, but without any associated fees. We will also explore using some of its rulesets and how to make changes to them yourself.
Running Falco
Following true Cloud Native methodology, we will use a container image to spawn Falco. That said, there are Linux rpm, deb, and binary files that you can install or execute directly, too, which appears to be the preferred route for their installation.
You can run Falco either on a host or by a userland container that additionally needs to access a pre-installed driver on the underlying host. Falco works by tapping into the kernel with elevated permissions to pick up the kernel's system calls (syscalls), and the driver is needed to offer that required functionality. We also need to provide Falco with the requisite permissions to enable such functionality. As described in Chapter 1, “What Is A Container?,” for a container runtime we define these permissions using kernel capabilities. To get an idea of what is available, you could do worse than looking over some of the names of the kernel capabilities in the manual (using the command man capabilities). Various versions of the manual are online too, such as this:
man7.org/linux/man-pages/man7/capabilities.7.html
To protect the underlying host, we will run Falco with as few privileges as possible. Be warned, however, that you will need a kernel version of v5.8 or higher to make use of the extended Berkeley Packet Filter (eBPF) driver without running a one-off --privileged container to install that driver to the underlying host(s) that Falco will run on. The Berkeley Packet Filter has been extended to allow increased access to the networking stack to applications via the kernel.
If you are lucky enough to have a kernel of v5.8 or later, the way around the one-off driver installation is to add the CAP_SYS_BPF option to your running container at startup time, which the more modern kernels will support. Add it using this command-line switch:
--cap--add SYS_BPF
For this demonstration, we will not assume that you have that kernel version, so we will install the driver on a host where we will use the one-off container method. The commands are as follows:
$ docker pull falcosecurity/falco-driver-loader:latest $ docker run --rm -it --privileged -v /root/.falco:/root/.falco \ -v /proc:/host/proc:ro -v /boot:/host/boot:ro \ -v /lib/modules:/host/lib/modules:ro \ -v /usr:/host/usr:ro -v /etc:/host/etc:ro \ falcosecurity/falco-driver-loader:latest
As you can see, we are using the insecure --privileged switch to gain the elevated permissions required to install the Falco driver. Listing 3.1 shows part of the output from the command, in which Dynamic Kernel Module Support (DKMS) is called into action on Debian derivatives and a kernel module is used.
Listing 3.1: DKMS Assisting with the Privileged Kernel Module Installation
Building module: cleaning build area… make -j4 KERNELRELEASE=4.15.0-20-generic -C /lib/modules/4.15.0-20-generic/build
M=/var/lib/dkms/falco/85c88952b018fdbce246422[…snip]/build… cleaning build area… DKMS: build completed. falco.ko: Running module version sanity check. - Original module - No original module exists within this kernel - Installation - Installing to /lib/modules/4.15.0-20-generic/kernel/extra/
Although the kernel version (4.15.0.20-generic) seems like a long way off from version 5.8, around version v4.19 the versions jumped to v5.4. To check that the process has automatically loaded up the kernel module as hoped, we can run this lsmod command:
$ lsmod | grep falco falco 634880 0
As we can see, falco is present in the list of loaded modules, so we can continue to proceed. Obviously, if you installed packages directly onto the host as the root user, this step would not be needed, but it is important to illustrate that container protection security tools also have trade-offs, and suffice to say functionality like rootless mode will not accommodate such functionality without some heartache. Relinquishing an undefined security control, such as having a common attack vector across all hosts, to onboard a security tool to protect running containers is a necessary evil; in this case, the kernel module is essential to Falco's functionality. Be aware that you are allowing the tool to tap into the very lowest level of a host's innards (and its running containers), so you need to be completely sure that the security product to which you are offering privileged access is fully trustworthy. On a large, containerized estate, with orchestrators and potentially tens of thousands of running containers on differing varieties of hosts, the fact that you are adding another attack vector to each and every host in the estate needs to be carefully considered. You are effectively opening up a predictable security hole (that is, it is predictable if an attacker knows that a privileged container runs on each host) that can be exploited throughout the estate if a vulnerability is found.
Next, to run our Falco container, we will run the following long command all on one line ideally to enable the kernel capability CAP_SYS_PTRACE. According to the SYS_PTRACE man page (man7.org/linux/man-pages/man2/ptrace.2.html), we can control and manipulate other processes with this privilege as well as move data into the memory space of processes.
$ docker run --rm -it --security-opt apparmor:unconfined \ --cap-add SYS_PTRACE \ --pid=host $(ls /dev/falco* | xargs -I {} echo --device {}) -v
/var/run/docker.sock:/var/run/docker.sock \ falcosecurity/falco-no-driver:latest
Note that we're demonstrating Falco on a Linux Mint machine (which is based on Ubuntu 18.04), and this command uses AppArmor effectively to stop rogue processes accessing several locked-away parts of a system. To use it, we also need to add the following switch to provide the required permissions to our container:
--security-opt apparmor:unconfined
As demonstrated in Chapter 1, you might also recognize that the container is offered the ability to access the host's process table namespace with the --pid switch on the Docker command.
Think about this for a moment. From a security vendor's perspective, AppArmor has clearly
