Secure deployments - Docker & K8s

Setup

We will be using a virtual machine in the faculty's cloud.

When creating a virtual machine in the Launch Instance window:

Name your VM using the following convention: scgc_lab<no>_<username>, where <no> is the lab number and <username> is your institutional account.
Select Boot from image in Instance Boot Source section
Select SCGC Template in Image Name section
Select the g.medium flavor.

In the base virtual machine:

Download the laboratory archive from here in the work directory. Use: wget https://repository.grid.pub.ro/cs/scgc/laboratoare/lab-kubernetes.zip to download the archive.
Extract the archive.
Start the virtual machines using bash runvm.sh.
The username for connecting to the nested VMs is student and the password is student.

$ # change the working dir
$ cd ~/work
$ # download the archive
$ wget https://repository.grid.pub.ro/cs/scgc/laboratoare/lab-kubernetes.zip
$ unzip lab-kubernetes.zip
$ # start VMs; it may take a while
$ bash runvm.sh
$ # check if the VMs booted
$ virsh net-dhcp-leases labvms

Introduction

This lab is structured into 2 parts:

Learn how to securely deploy applications using Docker containers.
Learn how to deploy applications using Kubernetes.

Why Containers?

easy service install
isolated test environments
local replicas of production environments

Objectives

container management (start, stop, build)
service management
container configuration and generation

What are containers?

Containers are an environment in which we can run applications isolated from the host system.

In Linux-based operating systems, containers are run like an application which has access to the resources of the host station, but which may interact with processes from outside the isolated environment.

The advantage of using a container for running applications is that it can be easily turned on and off and modified. Thus, we can install applications in a container, configure them and run them without affecting the other system components

A real usecase where we run containers is when we want to set up a server that depends on fixed, old versions of certain libraries. We don't want to run that server on our system physically, as conflicts with other applications may occur. Containerizing the server, we can have a version of the library installed on the physical machine and another version installed on the container without conflict between them.

Containers versus virtual machines?

Both containers and virtual machines allow us to run applications in an isolated environment. However, there are fundamental differences between the two mechanisms. A container runs directly on top of the operating system. Meanwhile, a virtual machine runs its own kernel and then runs the applications on top of that. This added abstraction layer adds overhead to running the desired applications, and the overhead slows down the applications.

Another plus for running containers is the ability to build and pack them iteratively. We can easily download a container from a public repository, modify it, and upload it to a public repository without uploading the entire image. We can do that because changes to a container are made iteratively, saving the differences between the image original and modified version.

There are also cases where we want to run applications inside a virtual machine. E.g, if we want to run a compiled application for an operating system other than Linux, we could not do this because containers can run applications that are compiled for the system host operation. Virtual machines can also run operating systems other than the operating system host.

Docker

Starting a container

To start an application inside a Docker container use the following command:

student@lab-docker:~$ sudo docker run -it gitlab.cs.pub.ro:5050/scgc/cloud-courses/ubuntu:18.04 bash
Unable to find image 'ubuntu:18.04' locally
18.04: Pulling from library/ubuntu
11323ed2c653: Already exists
Digest: sha256:d8ac28b7bec51664c6b71a9dd1d8f788127ff310b8af30820560973bcfc605a0
Status: Downloaded newer image for ubuntu:18.04
root@3ec334aece37:/# cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.6 LTS"
root@3ec334aece37:/#

The docker command was run using the following parameters:

run, start a container;
-i, starts an" interactive "container, which accepts keyboard input;
-t, associates a terminal to the run command;
ubuntu: 18.04 is the name of the image we want to use. [Dockerhub] (https://hub.docker.com/) is a public image repository from which we can download already built images;
bash, the command we want to run in the container.

We can also run a non-interactive command in a container as follows:

student@lab-docker:~$ sudo docker run gitlab.cs.pub.ro:5050/scgc/cloud-courses/ubuntu:18.04 ps -ef
UID          PID    PPID  C STIME TTY          TIME CMD
root           1       0  0 12:01 ?        00:00:00 ps -ef

note

The ps -ef command would show all active processes in the system. We notice that only one command appears in the output above, because we are running in an isolated environment. We will return to this in the "Container Security" section.

However, we do not want to always run containers in the foreground. If we want to run a script that cannot be run in the host environment, and this script will run for a long time, we prefer to run the command in the background.

To start a container in the background, use the -d option for the docker run command as follows:

student@lab-docker:~$ sudo docker run -d gitlab.cs.pub.ro:5050/scgc/cloud-courses/ubuntu:18.04 sleep 10000
a63ee06826a33c0dfab825a0cb2032eee2459e0721517777ee019f59e69ebc02
student@lab-docker:~$ sudo docker ps
CONTAINER ID   IMAGE          COMMAND         CREATED         STATUS         PORTS     NAMES
a63ee06826a3   ubuntu:18.04   "sleep 10000"   7 seconds ago   Up 5 seconds             wonderful_lewin
student@lab-docker:~$ sudo docker exec -it a63ee06826a3 /bin/bash
root@a63ee06826a3:/# ps -ef
UID          PID    PPID  C STIME TTY          TIME CMD
root           1       0  0 02:19 ?        00:00:00 sleep 10000
root           7       0  2 02:19 pts/0    00:00:00 /bin/bash
root          19       7  0 02:20 pts/0    00:00:00 ps -ef
root@a63ee06826a3:/# exit

We can see that the container started by us is still running by running the docker ps command. Relevant columns

CONTAINER ID
NAMES

To connect to a container running in the background, use the docker exec command along with the container ID or name selected using the docker ps command:

student@lab-docker:~$ sudo docker exec -it a63ee06826a3 /bin/bash
root@a63ee06826a3:/# ps -ef
UID          PID    PPID  C STIME TTY          TIME CMD
root           1       0  0 02:19 ?        00:00:00 sleep 10000
root           7       0  2 02:19 pts/0    00:00:00 /bin/bash
root          19       7  0 02:20 pts/0    00:00:00 ps -ef
root@a63ee06826a3:/# exit

To stop a container running in the background, use the docker stop command along with the container ID or name as follows:

student@lab-docker:~$ sudo docker stop a63ee06826a3
a63ee06826a3
student@lab-docker:~$ sudo docker ps
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES
student@lab-docker:~$

Exercise: Starting a container

Start a container in the background based on the quay.io/rockylinux/rockylinux:8 image.
Connect to the container just turned on and run the yum install bind-utils command.
Disconnect from container.

Context: Container separation

Most of the time when we use containers we do not use them interactively. They have a well-defined purpose, to run a service, an application, or to do a set of fixed operations.

A constructive approach to using containers is do one thing and do it well. For this reason, we recommend that each container be built with a single purpose in mind.

For example, for a web application we might have the following approach:

a container running an http server;
a container running a database.

This architecture allows us to change a container, such as changing the type of database used without changing the entire container.

Building a container

Most times just running a container interactively and connecting to it when the need arises is not enough. We want a way to automatically build and distribute single-use containers. For example, we want to use purpose build containers when running a CI/CD system that build a website and publishes it to the web. Each website has its own setup requirements, and we'd like to automate this. We could add automation by running a script, but in this case we'd lose one of the positives of running containers, the iterative nature of images, because the docker images would be monolithic.

In order to create a container we need to define a Dockerfile file as follows:

FROM gitlab.cs.pub.ro:5050/scgc/cloud-courses/ubuntu:18.04

ARG DEBIAN_FRONTEND=noninteractive
ARG DEBCONF_NONINTERACTIVE_SEEN=true

RUN apt-get update
RUN apt-get install -y software-properties-common

RUN apt-get install -y firefox

Each line contains commands that will be interpreted by Docker when building the image:

FROM, specifies the base container image
RUN, runs in container

This container will then be used to compile a container which can run Firefox.

It should be noted that in the process of building containers we have to use non-interactive commands, because we do not have access to the terminal where the terminal is built, so we can not write the keyboard options.

To build the container we will use the following command:

student@lab-docker:~$ docker build -t firefox-container  .

When we run the command we base that the Dockerfile file is in the current directory (~). The -t option will generate a container image named firefox-container.

To list container images on the machine use the following command:

student@lab-docker:~$ docker image list

This list contains both internally downloaded and locally built containers.

Exercise: Generate a container image

Write a Dockerfile.rocky file containing a recipe for generating a container image based on the quay.io/rockylinux/rockylinux:8 container in which to install the bind-utils tool.

note

To generate a container using a file other than the default Dockerfile we use the -f option.

Start the container generated in the previous exercise and run the command nslookup hub.docker.com to verify the installation of the package.

Downloading containers

Another important principle, both in the use of containers and in programming in general, is reusability. Instead of developing a new solution for every problem we encounter, we can use a solution that has already been implemented and submitted to a public repository.

For example, if we want to use a MySQL database to store information, instead of using a basic Ubuntu container and installing and configuring the server ourselves, we can download a container that already has the package installed.

Running commands in an unloaded container

We will use as an example, a set of containers consisting of a MySQL database and a WordPress service.

To start the two containers we will use the following commands:

student@lab-docker:~$ sudo docker network remove test-net
test-net
student@lab-docker:~$ sudo docker network create test-net
69643d63f7a785c07d4b93cf77a8b921e97595da778344e9aa8f62ac9cb6909a
student@lab-docker:~$ sudo docker run -d --hostname db --network test-net -e "MYSQL_ROOT_PASSWORD=somewordpress" -e "MYSQL_DATABASE=wordpress" -e "MYSQL_USER=wordpress" -e "MYSQL_PASSWORD=wordpress" mysql:5.7
657e3c4a23e120adf0eb64502deead82e156e070f7e9b47eff522d430279d3e1
student@lab-docker:~$ sudo docker run -d --hostname wordpress --network test-net -p "8000:80" -e "WORDPRESS_DB_HOST=db" -e "WORDPRESS_DB_USER=wordpress" -e "WORDPRESS_DB_PASSWORD=wordpress" gitlab.cs.pub.ro:5050/scgc/cloud-courses/wordpress:latest
Unable to find image 'wordpress:latest' locally
latest: Pulling from library/wordpress
c229119241af: Pull complete
47e86af584f1: Pull complete
e1bd55b3ae5f: Pull complete
1f3a70af964a: Pull complete
0f5086159710: Pull complete
7d9c764dc190: Pull complete
ec2bb7a6eead: Pull complete
9d9132470f34: Pull complete
fb23ab197126: Pull complete
cbdd566be443: Pull complete
be224cc1ae0f: Pull complete
629912c3cae4: Pull complete
f1bae9b2bf5b: Pull complete
19542807523e: Pull complete
59191c568fb8: Pull complete
30be9b012597: Pull complete
bb41528d36dd: Pull complete
bfd3efbb7409: Pull complete
7f19a53dfc12: Pull complete
23dc552fade0: Pull complete
5133d8c158a7: Pull complete
Digest: sha256:df2edd42c943f0925d4634718d1ed1171ea63e043a39201c0b6cbff9d470d571
Status: Downloaded newer image for wordpress:latest
b019fd009ad4bf69a9bb9db3964a4d446e9681b64729ffb850af3421c1df070c

The useful options above are:

-e sets an environment variable. This variable will be received by the container;
-p exposes an internal port of the container ( 80) to a port on the host machine (8000);
--hostname makes it so the container uses a specific hostname;
--network connects the container to a network other than the default.

We noticed in the output that we created the test-net network. We did this because in the default docker configuration, containers cannot communicate between themselves

We can connect using the Firefox browser to the virtual machine on port 8000 to configure the WordPress server.

Exercise: Running commands in the container

Start a container that hosts the NextCloud file sharing service. To connect to the NextCloud service, you need to expose the HTTP server running in the virtual machine. To do this, follow the example above. The container image name is nextcloud.

Automate container startup using Docker Compose

As we can see from the above example, we can start containers using the docker run command, but that means running a command for each container. This is simple when we only need to start two containers, but if we want to start more than two containers, or if we want to offer users a "one click" solution and we have a suite of containers needed for our solution, running in an ordered manner for each container does not scale.

The solution to this issue is the Docker Compose mechanism. It allows an administrator to write a specification for a work environment, including options for running containers, volumes running containers, and networks where containers will communicate.

The command is called docker-compose, and it uses docker-compose.yaml files which look like this:

services:
  db:
    image: mysql:5.7
    networks:
      - wordpress-net
    environment:
      MYSQL_ROOT_PASSWORD: somewordpress
      MYSQL_DATABASE: wordpress
      MYSQL_USER: wordpress
      MYSQL_PASSWORD: wordpress

  wordpress:
    depends_on:
      - db
    image: wordpress:latest
    networks:
      - wordpress-net
    ports:
      - "8000:80"
    environment:
      WORDPRESS_DB_HOST: db
      WORDPRESS_DB_USER: wordpress
      WORDPRESS_DB_PASSWORD: wordpress
      WORDPRESS_DB_NAME: wordpress

networks:
    wordpress-net:

In order to start the containers we use the docker-compose up command:

student@lab-docker:~$ sudo docker-compose up
WARNING: Some networks were defined but are not used by any service: wordpress-net
Creating network "student_default" with the default driver
Creating student_db_1 ... done
Creating student_wordpress_1 ... done
Attaching to student_db_1, student_wordpress_1
db_1         | 2022-04-05 03:48:41+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 5.7.37-1debian10 started.
db_1         | 2022-04-05 03:48:41+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
db_1         | 2022-04-05 03:48:42+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 5.7.37-1debian10 started.
db_1         | 2022-04-05 03:48:42+00:00 [Note] [Entrypoint]: Initializing database files
db_1         | 2022-04-05T03:48:42.223165Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
db_1         | 2022-04-05T03:48:42.819383Z 0 [Warning] InnoDB: New log files created, LSN=45790
db_1         | 2022-04-05T03:48:42.931685Z 0 [Warning] InnoDB: Creating foreign key constraint system tables.
db_1         | 2022-04-05T03:48:43.011806Z 0 [Warning] No existing UUID has been found, so we assume that this is the first time that this server has been started. Generating a new UUID: 49a0ec32-b493-11ec-b38d-0242ac150002.
db_1         | 2022-04-05T03:48:43.019048Z 0 [Warning] Gtid table is not ready to be used. Table 'mysql.gtid_executed' cannot be opened.
wordpress_1  | WordPress not found in /var/www/html - copying now...
wordpress_1  | Complete! WordPress has been successfully copied to /var/www/html
wordpress_1  | No 'wp-config.php' found in /var/www/html, but 'WORDPRESS_...' variables supplied; copying 'wp-config-docker.php' (WORDPRESS_DB_HOST WORDPRESS_DB_NAME WORDPRESS_DB_PASSWORD WORDPRESS_DB_USER)
wordpress_1  | AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 172.21.0.3. Set the 'ServerName' directive globally to suppress this message
wordpress_1  | AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 172.21.0.3. Set the 'ServerName' directive globally to suppress this message
wordpress_1  | [Tue Apr 05 03:48:43.798334 2022] [mpm_prefork:notice] [pid 1] AH00163: Apache/2.4.53 (Debian) PHP/7.4.28 configured -- resuming normal operations
wordpress_1  | [Tue Apr 05 03:48:43.798714 2022] [core:notice] [pid 1] AH00094: Command line: 'apache2 -D FOREGROUND'
db_1         | 2022-04-05T03:48:44.339284Z 0 [Warning] A deprecated TLS version TLSv1 is enabled. Please use TLSv1.2 or higher.
db_1         | 2022-04-05T03:48:44.339352Z 0 [Warning] A deprecated TLS version TLSv1.1 is enabled. Please use TLSv1.2 or higher.
db_1         | 2022-04-05T03:48:44.339950Z 0 [Warning] CA certificate ca.pem is self signed.
db_1         | 2022-04-05T03:48:44.547479Z 1 [Warning] root@localhost is created with an empty password ! Please consider switching off the --initialize-insecure option.

Notice that both containers run in the foreground. In order to start the containers in the background, we have to user the -d option.

To stop the containers specified in the docker-compose.yaml file we use the docker-compose down command as follows:

student@lab-docker:~$ sudo docker-compose down
WARNING: Some networks were defined but are not used by any service: wordpress-net
Removing student_wordpress_1 ... done
Removing student_db_1        ... done
Removing network student_default

Exercise: Automation using Docker Compose

Write a docker-compose.yaml file that will automatically start the nextcloud container when running the docker-compose up command.

Using persistent storage in containers

When we work with applications that we install on a cluster, they store data ephemerally. Thus, when deleting the container, all the information in the container is deleted. We don't want this to happen in the example of a database, where we rely on information being stored for a long time.

To start a container to which we attach a storage volume, we start the container as follows:

student@lab-docker:~$ sudo docker run -d -v mysql-volume:/var/lib/mysql -e "MYSQL_ROOT_PASSWORD=somewordpress" -e "MYSQL_DATABASE=wordpress" -e "MYSQL_USER=wordpress" -e "MYSQL_PASSWORD=wordpress" mysql:5.7
07ae337cead33307e6146f4e7142345e59d59dd29334b6e37f47268b58d093ac
student@lab-docker:~$ sudo docker exec -it 07ae337cead33307e6146f4e7142345e59d59dd29334b6e37f47268b58d093ac /bin/bash
root@07ae337cead3:/# echo "Hello" > /var/lib/mysql/test-file
root@07ae337cead3:/# exit
student@lab-docker:~$ sudo docker stop 07ae337cead33307e6146f4e7142345e59d59dd29334b6e37f47268b58d093ac
07ae337cead33307e6146f4e7142345e59d59dd29334b6e37f47268b58d093ac
student@lab-docker:~$ sudo docker rm 07ae337cead33307e6146f4e7142345e59d59dd29334b6e37f47268b58d093ac
07ae337cead33307e6146f4e7142345e59d59dd29334b6e37f47268b58d093ac
student@lab-docker:~$ sudo docker run -d -v mysql-volume:/var/lib/mysql -e "MYSQL_ROOT_PASSWORD=somewordpress" -e "MYSQL_DATABASE=wordpress" -e "MYSQL_USER=wordpress" -e "MYSQL_PASSWORD=wordpress" mysql:5.7
ad1b42b46654a8d4c721e69e824aa7ee18f1e39a85e0b27f1ac966c355a2786a
student@lab-docker:~$ sudo docker exec -it ad1b42b46654a8d4c721e69e824aa7ee18f1e39a85e0b27f1ac966c355a2786a /bin/bash
root@ad1b42b46654:/# cat /var/lib/mysql/test-file
Hello

note

While docker stop stops the container from running, the container's data is pruned after running the docker rm command.

The -v option attaches the mysql-volume to the mysql container to the /var/lib/mysql path. We notice that after we connected the volume, we wrote "Hello" in a file and it could be read after we restarted the container. Volumes are defaulted to /var/lib/docker/volumes/.

If we want to mount a directory or file on the host system as persistent storage, we can do so using the path to the directory we want to use, instead of the volume name we want to use. . The following example illustrates this option:

student@lab-docker:~$ sudo docker run -d -v ~/mysql-vol/:/shared-dir -e "MYSQL_ROOT_PASSWORD=somewordpress" -e "MYSQL_DATABASE=wordpress" -e "MYSQL_USER=wordpress" -e "MYSQL_PASSWORD=wordpress" mysql:5.7
628de4f3c693b25396de4bbaa951636535ecb1c167b1cca785028479676b7cec
student@lab-docker:~$ sudo docker exec -it 628de4f3c693b25396de4bbaa951636535ecb1c167b1cca785028479676b7cec /bin/bash
root@628de4f3c693:/# cat /shared-dir/test-file
Hello

In the case of containers that are run by docker-compose, a volume-type entry will look like this:

services:
  db:
    image: mysql:5.7
    networks:
      - wordpress-net
    volumes:
      - mysql-vol:/var/lib/mysql
    environment:
      MYSQL_ROOT_PASSWORD: somewordpress
      MYSQL_DATABASE: wordpress
      MYSQL_USER: wordpress
      MYSQL_PASSWORD: wordpress

  wordpress:
    depends_on:
      - db
    image: wordpress:latest
    networks:
      - wordpress-net
    ports:
      - "8000:80"
    environment:
      WORDPRESS_DB_HOST: db
      WORDPRESS_DB_USER: wordpress
      WORDPRESS_DB_PASSWORD: wordpress
      WORDPRESS_DB_NAME: wordpress

volumes:
    mysql-vol:

networks:
    wordpress-net:

note

Note that when we run the docker-compose down command, the volume defined in docker-compose.yaml is deleted. In order not to delete the volumes from the recipe, we need to run the docker-compose stop command to stop the containers defined in the YAML file.

Exercise: Mount a persistent volume in the container

Start a container from the nextcloud image to which you attach a volume called nextcloud-vol to /var/www/html. Restart the container and check that the configurations made when starting the container have been saved.

Container security

An advantage of using containers, in addition to the ease of building and starting containers, comes from the fact that a container runs in an isolated environment from the rest of the system. From this we can limit the running of applications in the container. We can do this by limiting process access to other processes in the system, as we saw in the example bellow, or we can do this by limiting the number of cyclesCPUs that can be accessed by the container, or by limiting the ram memory that can be allocated by applications in the container.

However, a disadvantage that containers have over virtual machines comes from the fact that a container runs on the same system as the host system and when it makes system calls it runs code from within the host kernel. If a vulnerability is discovered that allows an application to exit the container, it can affect the entire system, especially if it is a vulnerability at the kernel level. But in the case of a viral machine, a machine runs in an environment completely isolated from the physical system, so it has no way to receive additional access to system resources.

Why Kubernetes?

Container solutions (like Docker) are very good for deploying applications in predictable and isolated environments. However, for large systems, with a lot of nodes, it is impractical to manage containers with docker commands.

For this, container orchestration solutions have been developed. They manage a pool of worker nodes (basically, Linux machines with Docker installed) and containers are dynamically allocated to those nodes. The orchestrator also takes care of container lifecycle operations (start, stop, scale, upgrade etc.).

Examples of container orchestration solutions:

Kubernetes
HashiCorp Nomad
Docker Swarm
RedHat OpenShift (based on Kubernetes)

For this lab, we will be focusing on Kubernetes.

Creating a Kubernetes cluster

In Kubernetes, the terminology for the pool of worker nodes is Kubernetes cluster.

There are many approaches for deploying and installing a Kubernetes cluster, ranging from single-node solutions suitable for testing and development to full-scale, production-ready clusters.

For this lab, we will be using kind (acronym for Kubernetes in Docker), which deploys a lightweight, single-node cluster, inside Docker.

note

kind is already installed on the lab machine. If you want to know how to install it on a different machine, check out the user guide.

For creating a cluster on the lab machine, use the kind create cluster command:

student@lab-kubernetes:~$ kind create cluster
Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.23.4) 🖼
 ✓ Preparing nodes 📦
 ✓ Writing configuration 📜
 ✓ Starting control-plane 🕹️
 ✓ Installing CNI 🔌
 ✓ Installing StorageClass 💾
Set kubectl context to "kind-kind"
You can now use your cluster with:

kubectl cluster-info --context kind-kind

Thanks for using kind! 😊

Kubernetes CLI

The official tool for interacting with a Kubernetes cluster is kubectl.

note

kubectl is already installed on the lab machine. If you want to know how to install it on a different machine, check out the documentation.

Use kubectl cluster-info to show information about the cluster you deployed. You will see that the cluster is running locally:

student@lab-kubernetes:~$ kubectl cluster-info
Kubernetes control plane is running at https://127.0.0.1:41821
CoreDNS is running at https://127.0.0.1:41821/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

Use kubectl get nodes to show information about the cluster nodes. You will see just a single node:

student@lab-kubernetes:~$ kubectl get nodes
NAME                 STATUS   ROLES                  AGE   VERSION
kind-control-plane   Ready    control-plane,master   19h   v1.23.4

Pods

The basic resource in Kubernetes is the pod which typically encapsulates a container with the resources it needs (e.g. config files, volumes etc.).

In some usecases, a pod can contain multiple containers (also called sidecar containers). We won't be addressing this in the lab.

Launching a pod

Launching a pod is very similar to launching a Docker container. We will use the kubectl run command to do that.

We will use the gitlab.cs.pub.ro:5050/scgc/cloud-courses/hello-app:1.0 image, which is a simple HTTP server that echoes a message when receiving a request.

student@lab-kubernetes:~$ kubectl run hello-app --image=gitlab.cs.pub.ro:5050/scgc/cloud-courses/hello-app:1.0
pod/hello-app created

Getting information about a pod

For displaying a summary about pods or a certain pod, we can use kubectl get pods:

student@lab-kubernetes:~$ kubectl get pods
NAME                         READY   STATUS    RESTARTS   AGE
hello-app                    1/1     Running   0          12s

For detailed information, we can use kubectl describe:

student@lab-kubernetes:~$ kubectl describe pods hello-app
Name:         hello-app
Namespace:    default
Priority:     0
Node:         kind-control-plane/172.18.0.2
Start Time:   Fri, 08 Apr 2022 09:43:55 +0000
Labels:       run=hello-app
Annotations:  <none>
Status:       Running
IP:           10.244.0.89
[...]

Running commands inside a pod

For debugging purposes, we can enter a pod and run commands, using kubectl exec. This is similar to docker exec.

We will test that the container is working, by sending a request to its own HTTP endpoint:

student@lab-kubernetes:~$ kubectl exec -it hello-app -- /bin/sh
/ # wget -q -O - localhost:8080
Hello, world!
Version: 1.0.0
Hostname: hello-app
/ # exit

Getting logs from a pod

Similar to Docker, you can view the logs from a pod, using kubectl logs:

student@lab-kubernetes:~$ kubectl logs hello-app
2022/04/08 13:36:58 Server listening on port 8080
2022/04/08 13:37:34 Serving request: /

Removing a pod

A pod is removed with the kubectl delete command:

student@lab-kubernetes:~$ kubectl delete pods hello-app
pod "hello-app" deleted

Deployments

In many use cases, we want to describe the state of an application declaratively, so managing individual pods is not very convenient. Also, if an individual pod crashes or is deleted, it will not be respawned by default.

For this, we will use a deployment resource, which is an abstraction that encapsulates one or more pods.

Creating a deployment

Let's create a deployment for hello-app using the kubectl create command:

student@lab-kubernetes:~$ kubectl create deployment hello-app --image=gitlab.cs.pub.ro:5050/scgc/cloud-courses/hello-app:1.0
deployment.apps/hello-app created

We can see that the deployment is created, along with a pod:

student@lab-kubernetes:~$ kubectl get deployments
NAME        READY   UP-TO-DATE   AVAILABLE   AGE
hello-app   1/1     1            1           35s

student@lab-kubernetes:~$ kubectl get pods
NAME                         READY   STATUS    RESTARTS   AGE
hello-app-79df7f8b96-mn6wj   1/1     Running   0          39s

Getting information about a deployment

We can use kubectl describe to get details about a deployment:

student@lab-kubernetes:~$ kubectl describe deployments hello-app
Name:                   hello-app
Namespace:              default
CreationTimestamp:      Fri, 08 Apr 2022 12:40:55 +0000
Labels:                 app=hello-app
Annotations:            deployment.kubernetes.io/revision: 1
Selector:               app=hello-app
Replicas:               1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
[...]

Removing a deployment

A deployment is removed with the kubectl delete command:

student@lab-kubernetes:~$ kubectl delete deployments hello-app
deployment.apps "hello-app" deleted

Kubernetes manifests

Rather than using kubectl create commands, it is more convenient to use Kubernetes manifests. These are .yaml files that describe the resources we want to create. We can then create the resources with kubectl apply.

For example, let's define a manifest for creating the hello-app deployment:

student@lab-kubernetes:~$ cat hello-app-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello-app
  labels:
    app: hello
spec:
  replicas: 1
  selector:
    matchLabels:
      app: hello
  template:
    metadata:
      labels:
        app: hello
    spec:
      containers:
      - name: hello-app
        image: gitlab.cs.pub.ro:5050/scgc/cloud-courses/hello-app:1.0
        ports:
        - containerPort: 8080

Apply the manifest and check that the deployment and the pod was created:

student@lab-kubernetes:~$ kubectl apply -f hello-app-deployment.yaml
deployment.apps/hello-app created

student@lab-kubernetes:~$ kubectl get deployments
NAME        READY   UP-TO-DATE   AVAILABLE   AGE
hello-app   1/1     1            1           13s

student@lab-kubernetes:~$ kubectl get pods
NAME                         READY   STATUS    RESTARTS   AGE
hello-app-599bb4bf7f-l45k4   1/1     Running   0          17s

Exposing an app

Even if hello-app is deployed, there is currently no way of communicating with it from outside the cluster. The only way would be to use kubectl exec to enter the pod and communicate via localhost, which is not convenient.

For exposing the app outside the cluster, we need to create a Kubernetes service. This will act like a port-forwarding rule.

Creating a service

We can create a service using kubectl expose or using a manifest. We will choose the second option.

Let's define a service manifest and apply it:

student@lab-kubernetes:~$ cat hello-app-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: hello-app
spec:
  type: NodePort
  selector:
    app: hello
  ports:
    - protocol: TCP
      port: 8080
      targetPort: 8080
      nodePort: 30888

student@lab-kubernetes:~$ kubectl apply -f hello-app-service.yaml
service/hello-app created

student@lab-kubernetes:~$ kubectl get services
NAME         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
hello-app    NodePort    10.96.186.102   <none>        8080:30888/TCP   7m42s
kubernetes   ClusterIP   10.96.0.1       <none>        443/TCP          20h

note

There are multiple attributes that describe ports:

targetPort is the port that the pod listens to
port is the port that other pods from within the cluster can connect to the service
nodePort is the port that we can connect to from outside the cluster (must be between 30000-32767)

Connecting to a service

Before connecting to the service, we must determine the node's IP address:

student@lab-kubernetes:~$ kubectl describe nodes kind-control-plane | grep InternalIP
  InternalIP:  172.18.0.2

and then connect via curl:

student@lab-kubernetes:~$ curl http://172.18.0.2:30888
Hello, world!
Version: 1.0.0
Hostname: hello-app-599bb4bf7f-l45k4

Scaling an app

If the traffic to our app increases, we may need to scale the app (create mode pods, identical to the ones that already exist).

For example, let's scale hello-app to 10 pods. For this, change the value for replicas in hello-app-deployment.yaml to 10, and reapply the manifest:

student@lab-kubernetes:~$ kubectl apply -f hello-app-deployment.yaml
deployment.apps/hello-app configured

student@lab-kubernetes:~$ kubectl get pods
NAME                         READY   STATUS              RESTARTS   AGE
hello-app-599bb4bf7f-25w8g   1/1     Running             0          6s
hello-app-599bb4bf7f-7xzgr   0/1     ContainerCreating   0          5s
hello-app-599bb4bf7f-gr9xb   1/1     Running             0          6s
hello-app-599bb4bf7f-l45k4   1/1     Running             0          44m
hello-app-599bb4bf7f-mbgx7   0/1     ContainerCreating   0          6s
hello-app-599bb4bf7f-ps2dj   1/1     Running             0          6s
hello-app-599bb4bf7f-r6xqv   1/1     Running             0          6s
hello-app-599bb4bf7f-rrnws   0/1     ContainerCreating   0          5s
hello-app-599bb4bf7f-tnqtz   1/1     Running             0          6s
hello-app-599bb4bf7f-wh7qx   0/1     ContainerCreating   0          6s

After a while, you'll see that all 10 pods are running. Also, the deployment shows 10 available pods:

student@lab-kubernetes:~$ kubectl get deployments
NAME        READY   UP-TO-DATE   AVAILABLE   AGE
hello-app   10/10   10           10          45m

Replica sets

What actually happened is that a Kubernetes replica set associated with the deployment, of scale 10, was created:

student@lab-kubernetes:~$ kubectl get replicasets
NAME                   DESIRED   CURRENT   READY   AGE
hello-app-599bb4bf7f   10        10        10      1m

Testing the scaled app

Connect multiple times to the service, using curl. You will notice that each time, a different pod responds:

student@lab-kubernetes:~$ curl http://172.18.0.2:30888
Hello, world!
Version: 1.0.0
Hostname: hello-app-599bb4bf7f-r6xqv
student@lab-kubernetes:~$ curl http://172.18.0.2:30888
Hello, world!
Version: 1.0.0
Hostname: hello-app-599bb4bf7f-gr9xb
student@lab-kubernetes:~$ curl http://172.18.0.2:30888
Hello, world!
Version: 1.0.0
Hostname: hello-app-599bb4bf7f-rrnws
student@lab-kubernetes:~$ curl http://172.18.0.2:30888
Hello, world!
Version: 1.0.0
Hostname: hello-app-599bb4bf7f-7xzgr
student@lab-kubernetes:~$ curl http://172.18.0.2:30888
Hello, world!
Version: 1.0.0
Hostname: hello-app-599bb4bf7f-ps2dj

Upgrades and rollbacks

Deploying a different image version is done via editing the manifest and modifying the image field.

Upgrading an app

Update hello-app-deployment.yaml and change the image tag to 2.0. Then, redeploy the manifest:

student@lab-kubernetes:~$ kubectl apply -f hello-app-deployment.yaml
deployment.apps/hello-app configured

To follow the status of the update, use kubectl rollout status:

student@lab-kubernetes:~$ kubectl rollout status deployment hello-app
Waiting for deployment "hello-app" rollout to finish: 5 out of 10 new replicas have been updated...
[...]
deployment "hello-app" successfully rolled out

Run a curl to confirm that the upgraded application is running:

student@lab-kubernetes:~$ curl http://172.18.0.2:30888
Hello, world!
Version: 2.0.0
Hostname: hello-app-56c5b6c78b-74x9s

Rolling back

After the upgrade, a new replica set with scale 10 and the new image was created, and the old replica set was scaled down to 0:

student@lab-kubernetes:~$ kubectl get replicasets
NAME                   DESIRED   CURRENT   READY   AGE
hello-app-56c5b6c78b   10        10        10      5m55s
hello-app-599bb4bf7f   0         0         0       60m

For quickly reverting to the previous version (for example, in case of an error), we can use kubectl rollout undo:

student@lab-kubernetes:~$ kubectl rollout undo deployment hello-app
deployment.apps/hello-app rolled back

Confirm that the rollback was successful:

student@lab-kubernetes:~$ curl http://172.18.0.2:30888
Hello, world!
Version: 1.0.0
Hostname: hello-app-599bb4bf7f-fcsf2

Persistent storage

Most applications require persistent storage for keeping their state. For example, web servers need to store the content they are serving.

In the following steps, we will deploy an nginx application that will serve a custom-defined index.html.

Defining a ConfigMap

Kubernetes ConfigMaps are objects that can store arbitrary strings, including files.

Let's create a manifest that defines a ConfigMap that stores a custom index.html file. Note that the file content is defined inline:

student@lab-kubernetes:~$ cat nginx-html.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-html
data:
 index.html: |
    <html><body>Hello from SCGC Lab!</body></html>

Apply the manifest:

student@lab-kubernetes:~$ kubectl apply -f nginx-html.yaml
configmap/nginx-html created

Defining a Volume for a Deployment

Next, we will define an nginx deployment that mounts the ConfigMap by using a Volume.

student@lab-kubernetes:~$ cat nginx-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: gitlab.cs.pub.ro:5050/scgc/cloud-courses/nginx:latest
        ports:
        - containerPort: 80
        volumeMounts:
        - name: nginx-html-vol
          mountPath: "/usr/share/nginx/html/index.html"
          subPath: "index.html"
      volumes:
      - name: nginx-html-vol
        configMap:
          name: nginx-html
          items:
          - key: "index.html"
            path: "index.html"

note

Observe the following:

we defined a Volume called nginx-html-vol that takes its content from nginx-html ConfigMap
the volume is mounted in the nginx container, under /usr/share/nginx/html/index.html

Apply the manifest:

student@lab-kubernetes:~$ kubectl apply -f nginx-deployment.yaml
deployment.apps/nginx created

Also, expose the app via a service:

student@lab-kubernetes:~$ cat nginx-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: nginx
spec:
  type: NodePort
  selector:
    app: nginx
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
      nodePort: 30888

student@lab-kubernetes:~$ kubectl apply -f nginx-service.yaml
service/nginx created

Test that the app was correctly configured:

student@lab-kubernetes:~$ curl http://172.18.0.2:30888
<html><body>Hello from SCGC Lab!</body></html>

Communicating between apps

Apps deployed in Kubernetes can also communicate with each other, using the service names.

For showing this, we will configure the nginx app, so that for requests on /hello, it proxies the request to the hello-app service.

Creating the ConfigMap

We will need to create a ConfigMap for the custom nginx config file:

student@lab-kubernetes:~$ cat nginx-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-conf
data:
  default.conf: |
    server {
      listen       80;
      server_name  localhost;


      location / {
        root   /usr/share/nginx/html;
        index  index.html index.htm;
      }

      location /hello {
        proxy_pass http://hello-app:8080;
      }
    }

student@lab-kubernetes:~$ kubectl apply -f nginx-config.yaml
configmap/nginx-conf created

Mounting the config file

Modify the nginx deployment so that the config file is mounted in /etc/nginx/conf.d/default.conf:

student@lab-kubernetes:~$ cat nginx-deployment.yaml
[...]
        volumeMounts:
[...]
        - name: nginx-conf-vol
          mountPath: "/etc/nginx/conf.d/default.conf"
          subPath: "default.conf"
      volumes:
[...]
      - name: nginx-conf-vol
        configMap:
          name: nginx-conf
          items:
          - key: "default.conf"
            path: "default.conf"

student@lab-kubernetes:~$ kubectl apply -f nginx-deployment.yaml
deployment.apps/nginx configured

Testing the app

Test that requests on / work as before, but requests on /hello are proxied:

student@lab-kubernetes:~$ curl http://172.18.0.2:30888
<html><body>Hello from SCGC Lab!</body></html>

student@lab-kubernetes:~$ curl http://172.18.0.2:30888/hello
Hello, world!
Version: 1.0.0
Hostname: hello-app-599bb4bf7f-dxqxs

note

This example was only a didactical one, for showing how a config file can be mounted into a pod. For request routing, Kubernetes has a native mechanism, called Ingress

Namespaces

Even if containers represent isolated environments, we may need a broader isolation, for security purposes.

For examples, we may want to separate the applications of different customers, or development and production environments.

In Kubernetes, this is achieved using namespaces.

Listing namespaces

All the exercises until now were performed in the default namespace. But Kubernetes has several namespaces out of the box:

student@lab-kubernetes:~$ kubectl get namespaces
NAME                 STATUS   AGE
default              Active   25h
kube-node-lease      Active   25h
kube-public          Active   25h
kube-system          Active   25h
local-path-storage   Active   25h

For example, the kube-system namespace is used for Kubernetes internal resources, that should not be modified by the user:

student@lab-kubernetes:~$ kubectl get pods -n kube-system
NAME                                         READY   STATUS    RESTARTS   AGE
coredns-64897985d-6qnmw                      1/1     Running   0          25h
coredns-64897985d-f6k2t                      1/1     Running   0          25h
etcd-kind-control-plane                      1/1     Running   0          25h
kindnet-tbmt8                                1/1     Running   0          25h
kube-apiserver-kind-control-plane            1/1     Running   0          25h
kube-controller-manager-kind-control-plane   1/1     Running   0          25h
kube-proxy-dpz24                             1/1     Running   0          25h
kube-scheduler-kind-control-plane            1/1     Running   0          25h

Creating a new namespace

We can create a new namespace using kubectl create:

student@lab-kubernetes:~$ kubectl create namespace test
namespace/test created

Verifying namespace isolation

Create a simple nginx pod in the test namespace. Notice the -n test parameter.

student@lab-kubernetes:~$ kubectl run nginx --image=gitlab.cs.pub.ro:5050/scgc/cloud-courses/nginx:latest -n test
pod/nginx created

student@lab-kubernetes:~$ kubectl get pods -n test
NAME    READY   STATUS    RESTARTS   AGE
nginx   1/1     Running   0          7s

Connect to the pod and verify if the name of the hello-app service from the default namespace can be resolved:

student@lab-kubernetes:~$ kubectl exec -it nginx -n test -- /bin/bash
root@nginx:/# curl http://hello-app:8080
curl: (6) Could not resolve host: hello-app

note

The default namespace isolation is not very strong, because resources can still be accessed by FQDN or by IP address. But additional security can be implemented, such as denying all network traffic between namespaces.

Setup​

Introduction​

Why Containers?​

Objectives​

What are containers?​

Containers versus virtual machines?​

Docker​

Starting a container​

Exercise: Starting a container​

Context: Container separation​

Building a container​

Exercise: Generate a container image​

Downloading containers​

Running commands in an unloaded container​

Exercise: Running commands in the container​

Automate container startup using Docker Compose​

Exercise: Automation using Docker Compose​

Using persistent storage in containers​

Exercise: Mount a persistent volume in the container​

Container security​

Why Kubernetes?​

Creating a Kubernetes cluster​

Kubernetes CLI​

Pods​

Launching a pod​

Getting information about a pod​

Running commands inside a pod​

Getting logs from a pod​

Removing a pod​

Deployments​

Creating a deployment​

Getting information about a deployment​

Removing a deployment​

Kubernetes manifests​

Exposing an app​

Creating a service​

Connecting to a service​

Scaling an app​

Replica sets​

Testing the scaled app​

Upgrades and rollbacks​

Upgrading an app​

Rolling back​

Persistent storage​

Defining a ConfigMap​

Defining a Volume for a Deployment​

Communicating between apps​

Creating the ConfigMap​

Mounting the config file​

Testing the app​

Namespaces​

Listing namespaces​

Creating a new namespace​

Verifying namespace isolation​

Setup

Introduction

Why Containers?

Objectives

What are containers?

Containers versus virtual machines?

Docker

Starting a container

Exercise: Starting a container

Context: Container separation

Building a container

Exercise: Generate a container image

Downloading containers

Running commands in an unloaded container

Exercise: Running commands in the container

Automate container startup using Docker Compose

Exercise: Automation using Docker Compose

Using persistent storage in containers

Exercise: Mount a persistent volume in the container

Container security

Why Kubernetes?

Creating a Kubernetes cluster

Kubernetes CLI

Pods

Launching a pod

Getting information about a pod

Running commands inside a pod

Getting logs from a pod

Removing a pod

Deployments

Creating a deployment

Getting information about a deployment

Removing a deployment

Kubernetes manifests

Exposing an app

Creating a service

Connecting to a service

Scaling an app

Replica sets

Testing the scaled app

Upgrades and rollbacks

Upgrading an app

Rolling back

Persistent storage

Defining a ConfigMap

Defining a Volume for a Deployment

Communicating between apps

Creating the ConfigMap

Mounting the config file

Testing the app

Namespaces

Listing namespaces

Creating a new namespace

Verifying namespace isolation