What is Apparmor and how to add a security layer with it in Docker?

include <tunables/home>

# profile name
/usr/bin/myapp {
    # ---- 1: include -------
    include <file>
    include "my/sub/rule"

    # ---- 2: capability ----
    capability setuid,
    capability setgid,

    # ---- 3: network -------
    network inet dgram,
    network inet stream,

    # ---- 4: rlimit --------
    rlimit stack >= 5K,

    # ---- 5: file ----------
    @{HOME}/myfile rw,
    @{HOME}/app    ix,
}

$ aa-status
apparmor module is loaded.
54 profiles are loaded.
50 profiles are in enforce mode.
[...]
4 profiles are in complain mode.
[...]
2 processes have profiles defined.
2 processes are in enforce mode.
[...]
0 processes are in complain mode.
0 processes are unconfined but have a profile defined.

Interface AppArmor with Docker containers

How to use AppArmor with Docker?

On systems hardened with AppArmor, Docker can enforce AppArmor profiles on containers. By default, it automatically generates and applies a profile for containers named docker-default that is created in tmpfs and then loaded in the kernel. However, this behavior can be overridden by specifying a custom profile (already loaded in the kernel) with the flag --security-opt in the docker run command:

$ docker run --security-opt apparmor=<profile> <image>

⚠️The default profile is very permissive, if you want to harden the security of your application you most likely want to override it.

Profiles are applied on containers and not on the Docker Engine daemon. If you want to use a profile on the daemon, you can generate and load one by using AppArmor directly. A profile is available in the Docker Engine source repository for the daemon but it is currently not installed with Docker, it has to be loaded manually.

Tutorial

This section is a small tutorial presenting how to create profiles for containers with bane and how to apply them. It will go through the generation of a profile for a Node.js application.

Install AppArmor and bane

Make sure the AppArmor module is installed on your kernel with the command aa-status. If not, see distribution-specific instructions on how to install it:

• It is shipped by default on all Debian-based Linux distributions.
• Follow the wiki instructions to install it on Archlinux / Manjaro

To install bane, follow the instructions given on the release page of the tool depending on your OS and your computer architecture.

Generate an AppArmor profile for a docker container with bane

bane is a profile generator for docker containers. It simplifies the writing of profiles for docker containers. To start, we will create a very restricted profile preventing writing, network access, and use of any Linux capabilities. Then, to understand the rights our application requires, we will put this profile in audit and complain modes.

bane generates AppArmor profiles from .toml configuration files. Let's create a file named nodejs.toml with the following configuration:

# name of the profile, auto prefix with "docker-"
# so the final name will be "docker-nodejs"
Name = "nodejs"

[Filesystem]
# read only paths for the container
ReadOnlyPaths = [
	"/bin/**",
	"/boot/**",
	"/dev/**",
	"/etc/**",
	"/home/**",
	"/lib/**",
	"/lib64/**",
	"/media/**",
	"/mnt/**",
	"/opt/**",
	"/proc/**",
	"/root/**",
	"/sbin/**",
	"/srv/**",
	"/tmp/**",
	"/sys/**",
	"/usr/**",
]

[Network]
# deny raw sockets
Raw = false
# deny packet sockets which allow to manage raw packets
# at the device driver level
Packet = false
Protocols = [
	"tcp",
]

Now that we have a configuration file, we can generate our AppArmor profile:

$ sudo bane nodejs.toml

The generated profile is located in the file /etc/apparmor.d/containers/docker-nodejs. You should have the following profile:

#include <tunables/global>


profile docker-nodejs flags=(attach_disconnected,mediate_deleted) {
  #include <abstractions/base>

  network inet tcp,

  deny network raw,

  deny network packet,

  file,
  umount,

  deny /bin/** wl,
  deny /boot/** wl,
  deny /dev/** wl,
  deny /etc/** wl,
  deny /home/** wl,
  deny /lib/** wl,
  deny /lib64/** wl,
  deny /media/** wl,
  deny /mnt/** wl,
  deny /opt/** wl,
  deny /proc/** wl,
  deny /root/** wl,
  deny /sbin/** wl,
  deny /srv/** wl,
  deny /tmp/** wl,
  deny /sys/** wl,
  deny /usr/** wl,

  deny @{PROC}/* w,   # deny write for all files directly in /proc (not in a subdir)
  deny @{PROC}/{[^1-9],[^1-9][^0-9],[^1-9s][^0-9y][^0-9s],[^1-9][^0-9][^0-9][^0-9]*}/** w,
  deny @{PROC}/sys/[^k]** w,  # deny /proc/sys except /proc/sys/k* (effectively /proc/sys/kernel)
  deny @{PROC}/sys/kernel/{?,??,[^s][^h][^m]**} w,  # deny everything except shm* in /proc/sys/kernel/
  deny @{PROC}/sysrq-trigger rwklx,
  deny @{PROC}/mem rwklx,
  deny @{PROC}/kmem rwklx,
  deny @{PROC}/kcore rwklx,
  deny mount,
  deny /sys/[^f]*/** wklx,
  deny /sys/f[^s]*/** wklx,
  deny /sys/fs/[^c]*/** wklx,
  deny /sys/fs/c[^g]*/** wklx,
  deny /sys/fs/cg[^r]*/** wklx,
  deny /sys/firmware/efi/efivars/** rwklx,
  deny /sys/kernel/security/** rwklx,
}

Set the docker-nodejs profile to complain + audit modes

For that, edit the content of the file /etc/apparmor.d/containers/docker-nodejs which is the profile bane generated and replace the line:

profile docker-nodejs flags=(attach_disconnected,mediate_deleted)

with:

profile docker-nodejs flags=(audit,complain,attach_disconnected,mediate_deleted)

The flags we added tell AppArmor to load our profile in audit and complain modes.

In order to make the resulting configuration less permissive, we will remove all executing rights. To do that, replace the line:

file,

with:

/** rwmlk,

bane automatically loads profiles in AppArmor, but since we modified the docker-nodejs profile, we need to reload it with the command:

$ sudo apparmor_parser -r /etc/apparmor.d/containers/docker-nodejs

Start our container with the docker-nodejs AppArmor profile

Let's run a small Node.js application that runs a web server responding to requests with their content:

profile docker-nodejs flags=(audit,complain,attach_disconnected,mediate_deleted)

To access the server, we can use curl from the command line:

$ curl http://localhost:8000/test
{"code":"success","meta":{"total":0,"count":0},"payload":[]}%

Analyze the audit logs to understand which resources our web server needs to access

Now we will analyze logs produced by AppArmor to understand how our profile works. Shut down the docker container we just launch and open the AppArmor log file in /var/log/kern.log (or /var/log/audit/audit.log if you have auditd installed) with:

$ sudo tail -f /var/log/kern.log

Let's start our container again:

$ docker run --rm --security-opt apparmor=docker-nodejs -p 8000:8000 clementde/tutorial-docker-apparmor:v0.1.0

And run a curl command to request the web server:

$ curl http://localhost:8000/test

We can now analyze the logs to understand what our container needs to operate:

apparmor="AUDIT" operation="open" profil="docker-nodejs" name="/etc/ld.so.cache" pid=57949 comm="docker-entrypoi" requested_mask="r" fsuid=0 ouid=0
apparmor="AUDIT" operation="getattr" profil="docker-nodejs" name="/etc/ld.so.cache" pid=57949 comm="docker-entrypoi" requested_mask="r" fsuid=0 ouid=0
apparmor="AUDIT" operation="open" profil="docker-nodejs" name="/lib/x86_64-linux-gnu/libc-2.24.so" pid=57949 comm="docker-entrypoi" requested_mask="r" fsuid=0 ouid=0
apparmor="AUDIT" operation="getattr" profil="docker-nodejs" name="/lib/x86_64-linux-gnu/libc-2.24.so" pid=57949 comm="docker-entrypoi" requested_mask="r" fsuid=0 ouid=0
apparmor="AUDIT" operation="file_mmap" profil="docker-nodejs" name="/lib/x86_64-linux-gnu/libc-2.24.so" pid=57949 comm="docker-entrypoi" requested_mask="r" fsuid=0 ouid=0
apparmor="AUDIT" operation="open" profil="docker-nodejs" name="/usr/local/bin/docker-entrypoint.sh" pid=57949 comm="docker-entrypoi" requested_mask="r" fsuid=0 ouid=0
apparmor="AUDIT" operation="getattr" profil="docker-nodejs" name="/usr/local/bin/node" pid=57983 comm="docker-entrypoi" requested_mask="r" fsuid=0 ouid=0
apparmor="ALLOWED" operation="exec" profil="docker-nodejs" name="/usr/local/bin/node" pid=57949 comm="docker-entrypoi" requested_mask="x" denied_mask="x" fsuid=0 ouid=0 target="docker-nodejs//null-/usr/local/bin/node"
apparmor="ALLOWED" operation="file_mmap" profil="docker-nodejs//null-/usr/local/bin/node" name="/usr/local/bin/node" pid=57949 comm="node" requested_mask="rm" denied_mask="rm" fsuid=0 ouid=0
apparmor="ALLOWED" operation="file_mmap" profil="docker-nodejs//null-/usr/local/bin/node" name="/lib/x86_64-linux-gnu/ld-2.24.so" pid=57949 comm="node" requested_mask="rm" denied_mask="rm" fsuid=0 ouid=0

To understand the logs we got, note that:

• All lines with apparmor="AUDIT" tell us which authorized resources our container accessed.
• All lines with apparmor="ALLOWED" gives us the resources forbidden by our profile that our container accessed.

So to make our profile work, we need to give execute access to /usr/local/bin/node. To do that, we can add the following line to our profile in /etc/apparmor.d/containers/docker-nodejs:

/usr/local/bin/node ix,

We can now reload our profile with:

$ sudo apparmor_parser -r /etc/apparmor.d/containers/docker-nodejs

Let's give it a new try to see if everything works as expected. When we restart the container and make a curl request, new ALLOWED operations are potentially logged:

apparmor="ALLOWED" operation="accept" profil="docker-nodejs" pid=59178 comm="node" lport=8000 family="inet6" sock_type="stream" protocol=6 requested_mask="accept" denied_mask="accept"
apparmor="ALLOWED" operation="file_perm" profil="docker-nodejs" pid=59178 comm="node" laddr=::ffff:172.17.0.2 lport=8000 faddr=::ffff:172.17.0.1 fport=43368 family="inet6" sock_type="stream" protocol=6 requested_mask="receive" denied_mask="receive"
apparmor="ALLOWED" operation="file_perm" profil="docker-nodejs" pid=59178 comm="node" laddr=::ffff:172.17.0.2 lport=8000 faddr=::ffff:172.17.0.1 fport=43368 family="inet6" sock_type="stream" protocol=6 requested_mask="receive" denied_mask="receive"
apparmor="ALLOWED" operation="recvmsg" profil="docker-nodejs" pid=59178 comm="node" laddr=::ffff:172.17.0.2 lport=8000 faddr=::ffff:172.17.0.1 fport=43368 family="inet6" sock_type="stream" protocol=6 requested_mask="receive" denied_mask="receive"
apparmor="ALLOWED" operation="file_perm" profil="docker-nodejs" pid=59178 comm="node" laddr=::ffff:172.17.0.2 lport=8000 faddr=::ffff:172.17.0.1 fport=43368 family="inet6" sock_type="stream" protocol=6 requested_mask="send" denied_mask="send"
apparmor="ALLOWED" operation="file_perm" profil="docker-nodejs" pid=59178 comm="node" laddr=::ffff:172.17.0.2 lport=8000 faddr=::ffff:172.17.0.1 fport=43368 family="inet6" sock_type="stream" protocol=6 requested_mask="send" denied_mask="send"

Our container tries to communicate with IPv6, so we need to allow it in our profile by adding the line:

network inet6 tcp,

Set the profile to enforce mode

No more ALLOW operations are logged after reloading the profile and restarting our container. We can put it in enforce mode by removing the flags we added previously. Replace the line:

profile docker-nodejs flags=(audit,complain,attach_disconnected,mediate_deleted)

with:

profile docker-nodejs flags=(attach_disconnected,mediate_deleted)

When we reload the profile and restart the docker container, everything works as expected!

We can now check that our profile is well enforced. For that, we can open a shell inside our container and try to run commands:

$ docker exec -it $(docker ps -q) sh
# ls
sh: 1: ls: Permission denied
# whoami
sh: 2: whoami: Permission denied

Note that the profile is enforced only inside the container, which is why we can still open a shell with docker exec.

AppArmor in Kubernetes

Kubernetes allows you to load AppArmor profiles on containers. You can specify a profile to run a Pod container with, by adding an annotation to the Pod's metadata:

container.apparmor.security.beta.kubernetes.io/<container_name>: <profile_ref>

However, for the profile to be applied, the Pod must run on a node that supports AppArmor and that already has the right profile loaded in the kernel. To set up your profile on your nodes, you can either use a DaemonSet, a node initialization script, or SSH on your nodes to add it manually.

The documentation provides a great article on how to use AppArmor along with Kubernetes.

Conclusion

With AppArmor, you can strongly confine your Docker applications to greatly limit the impact of a potential compromise. Writing profiles may seem tedious, but with a little practice, you can easily write simple profiles that secure your containers and kubernetes applications.