Deployment

The main instance of CurryBO is hosted on a docker swarm on an ETH VM (chabicbvs001). All Reportcenter images this swarm come from GitLab CI/CD pipeline.

CI/CD

In order to build the images, a GitLab CI/CD pipeline is used. This pipeline executes the following steps, in this order:

• build: The docker image is built by running docker build
• push: The docker image is pushed to the docker registry hosted on the production server
• pull: The runner logs into the production server via SSH and updates the service (i.e. pulling the newest image and start it)

Docker Swarm

Docker Swarm is a docker mode that allows for a cluster size between standard docker and kubernetes, which makes it a very good tool for Reportcenter. Our cluster runs the following services (a service makes sure that as many replica of an image as configured are running at all times):

[currybo@chabicbvs001 ~]$ docker service ls
ID             NAME                        MODE         REPLICAS   IMAGE                                   PORTS

These services serve the following purposes:

• gitlab-runner-2: Compile docker images by picking up CI/CD jobs from GitLab
• nginx: Handles routing and forwards domains to their respecitve containers
• currybo-db: Database for CurryBO
• registry: Hosts built docker images.

nginx

Nginx is the service that listens on ports 80 (HTTP) and 443 (HTTPS) and forwards requests to the correct container. Nginx configuration is stored in /etc/nginx/conf.d and mounted into the nginx container. The nginx configuration for for currybo looks like this:

server {
	server_name reportcenter.mat.ethz.ch;
	client_max_body_size 100M;
	location / {
		resolver 127.0.0.11;
		proxy_pass http://reportcenter-matl:3000;
	}
	listen [::]:443 ssl http2;
	listen 443 ssl http2;
	ssl_certificate /etc/letsencrypt/live/reportcenter.mat.ethz.ch/fullchain.pem;
	ssl_certificate_key /etc/letsencrypt/live/reportcenter.mat.ethz.ch/privkey.pem;
}
 
server {
	listen 80 http2;
	listen [::]:80 http2;
 
	server_name reportcenter.mat.ethz.ch;
	client_max_body_size 100M;
 
	return 301 https://$host$request_uri;
	return 404;
}

This configuration

• listens to currybo.ethz.ch,
• forwards HTTP to HTTPS,
• uses he docker DNS resolver (127.0.0.11) to resolve docker swarm service names,
• Deencrypts the request (SSL termination), and
• forwards the request to the currybo service on port 3000 via HTTP.

The client_max_body_size 100M is important because users are allowed to upload pretty large files.

After updating nginx configuration, apply it by restarting the nginx container (--force makes sure that the new configuration is actually read):

docker service update --force nginx

SSL Certificates

We generate SSL certificate via Let’s Encrypt and their CLI certbot. In order to generate a new certificate,

• stop the nginx container by running

docker service update --replicas 0 nginx

This is necessary because certbot needs the 80 and 443 ports

• Generate the certificate with certbot

sudo certbot certonly

certbot will then ask if you want to start the certbot container. Type 1 (for “standalone”) and then type the domains the certificate should be generated for.

• Start the nginx container again

docker service update --replicas 1 nginx

The certificate are mounted into the nginx container, so no further configuration is necessary.

Renew certificates

In order to renew the certificates, a systemd timer ist used:

[Unit]
Description=Certbot Renewal
 
[Service]
ExecStart=/bin/bash /home/currybo/certificate-renew.sh

[Unit]
Description=Timer for Certbot Renewal
 
[Timer]
OnBootSec=300
OnUnitActiveSec=1w
 
[Install]
WantedBy=multi-user.target

#!/bin/bash
 
sudo certbot renew --pre-hook "docker service update --replicas 0 nginx" --post-hook "docker service update --replicas 1 nginx"

This will try to renew all certificates that would expire within a month

• When the system just booted, after 5min
• Every week otherwise

This systemd timer/service can be inspected with systemd status:

[currybo@chabicbvs001 ~]$ sudo systemctl status certbot-renew
○ certbot-renew.service - This service automatically renews any certbot certificates found
     Loaded: loaded (/usr/lib/systemd/system/certbot-renew.service; static)
     Active: inactive (dead) since Tue 2024-08-13 01:37:38 CEST; 7h ago
TriggeredBy: ● certbot-renew.timer
    Process: 672798 ExecStart=/usr/bin/certbot renew --noninteractive --no-random-sleep-on-renew $PRE_HOOK $POST_HOOK $RENEW_HOOK $DEPLOY_HOOK $C>
   Main PID: 672798 (code=exited, status=0/SUCCESS)
        CPU: 333ms

[currybo@chabicbvs001 ~]$ sudo systemctl status certbot-renew.timer
● certbot-renew.timer - This is the timer to set the schedule for automated renewals
     Loaded: loaded (/usr/lib/systemd/system/certbot-renew.timer; enabled; preset: disabled)
     Active: active (waiting) since Wed 2024-07-24 15:59:55 CEST; 2 weeks 5 days ago
      Until: Wed 2024-07-24 15:59:55 CEST; 2 weeks 5 days ago
    Trigger: Tue 2024-08-13 15:37:21 CEST; 6h left
   Triggers: ● certbot-renew.service