Let us discuss each of these concepts in detail in this article.
Immutable containers
Immutability allows us to build the infrastructure to an exact set of specifications and deploy the same container image in every environment. This is because immutable containers won’t be modified once they are started from an image. No changes are allowed, which include configuration changes, patches and updates to the application code. If any modifications such as code changes are needed, a new image must be built instead. This approach has several benefits such as rolling back by simply redeploying the old image in case of issues and better security as no unknown changes are not allowed on containers started from already verified images. Any changes must be done on a new version of the docker image. As the images go through changes, any known vulnerabilities or security issues can be fixed providing clean new containers again. One way to make containers immutable is to start them with –read-only flag but it should be noted that the application cannot write anything in the container.
Live scanning
Once docker images are built with security in mind and containers are spun up with appropriate hardening, another step in ensuring that the work loads running in the containers are safe is to perform live scanning on the running containers. This can uncover vulnerabilities that weren’t found in early stages such as when writing application code. In addition to it, monitoring the container traffic can also detect attacks attempted against the application running as well as the container itself. Application layer vulnerabilities such as SQL Injection, Cross Site Scripting, Remote Command Execution may be identified by using traditional web vulnerability scanners. The following figure shows automated scan results from OWASP ZAP run against an application running in a container.
Similarly, scanning container logs and analysing container network traffic can also give visibility into various attacks attempted. The following log entry from the container shows that a path traversal attack was attempted on the application running in the container. Even though the example discussed here was shown using a traditional web vulnerability scanner run from a different machine in the same network as the host, where the container is running, there are other approaches that can be used to perform live scanning. Some of the examples include:
Embedding an agent into the container that can perform the scans Side-car containers for log analysis Scanning by running the scanner as another container on the same host Scanning from the docker host (the host, where Docker engine is running) Scanning from a host in the same network as the docker host
Depending on certain conditions and choices, some of these options may or may not be feasible to implement. If these containers are built and run in CI/CD environments, security scans (DAST) can be performed immediately after a container is started right on the CI server. This will help detect the vulnerabilities even before the containers hosting the applications are in production. So, running scans from CI server is another option.
Baselines vs anomaly detection
Containers unlike traditional infrastructure (such as Virtual Machines) are lightweight and meant to run a single process. This form factor of containers increases the efficacy of anomaly detection by providing context around the applications that are running. This context of normal or expected behaviour is represented as baseline. The baseline allows us to perform effective monitoring of the container by looking for any deviations from the expected behaviour. Many runtime monitoring tools available for containers detect anomalies based on the system calls triggered from within the container. In addition to it, we can also perform log aggregation from the container to establish a baseline. So, aggregating logs and recording system calls triggered by the container while the application is running in its normal state are two common ways to establish a baseline for anomaly detection at a later stage. Logs extracted from the containers should be reviewed and analysed to understand the expected behaviour of the container. Logs obtained from docker native logging may not give a great visibility into the application specific activities. So, it is recommended to perform verbose logging and obtain more detailed logs possibly using a sidecar container. Once the expected behaviour is established, we will need to monitor the logs for any deviations from the expected behaviour. The following excerpt shows an example of a deviation in access logs showing a Local File Inclusion attack attempt. Similarly, the system calls should be recorded to establish a baseline for monitoring system calls in future. Operating Systems contain routines to perform various low level operations. If we want to invoke these operating system routines from userland programs, we need to invoke system calls. A System call is a bridge between the user program and Operating System routine. According to Wikipedia, “A system call is how a program requests a service from an operating system’s kernel. This may include hardware related services (e.g. accessing the hard disk), creating and executing new processes, and communicating with integral kernel services (like scheduling). System calls provide an essential interface between a process and the operating system.” The Falco project from sysdig is one of the most popular tools available for container runtime monitoring. It installs a Linux kernel module and monitors for malicious behaviours initiated from the containers. We can also customize Falco rules as we need. Let us see some Falco examples from it’s default ruleset. If someone reads sensitive files such as /etc/shadow from within a container using the following command, Falco detects it and leaves an alert. The following excerpt shows an alert from Falco. Clearly, the Falco alert shows that the command cat /etc/shadow is executed on an alpine container with the container id e8904b80c959. Similarly, if any files are written into sensitive directories such as /bin, which contains pre-installed binaries, it will result in an alert. The following command can be used to simulate it. The following excerpt shows an alert from Falco. Once again, the Falco alert shows that the command touch /bin/test is executed on an alpine container with the container id e8904b80c959.
Conclusions
Immutable containers are good but they may not be a good use case for every application. If any application requires to write changes to the container, immutability cannot be leveraged. Immutability can be planned during the design phase for newly developed applications, but it may be a hard choice to make for applications already built especially If the application needs to write changes within the containers. Live scanning comes with its own pros and cons. If we are running a web application security scan, all we need is the application URL and possibly the login details. On its downside, it may not find every single vulnerability and non-application layer attacks may require different scanning tools or agents. Anomaly detection using the established baselines is only as good as the baselines established. Incomplete and incorrect baselines can lead to false positives and/or false negatives.
Sources
https://falco.org/docs/getting-started/running/ https://cloud.google.com/solutions/best-practices-for-operating-containers https://www.sumologic.com/insight/mutable-immutable-infrastructure/
