Infrastructure Mint System

A short introduction into the infrastructure of Mint System.

Drivers

Early on we wanted to be able to self-host web application.

ERP-Systems are operating system for companies

  • They have to be hosted reliably
  • Companies want know where their data is

Infrastructure as Code:

  • Every system / server can be rebuilt from code
  • The deployment of configuration must be automated

Run apps with Docker containers:

  • Manage containers not apps
  • Every decent web app provides Docker images

Hardware

We don't want to manage hardware. We use these providers:

  • Hetzner
  • Exoscale
  • Infomaniak
  • Ungleich

And service partners.

Ansible

Everything is done with Ansible. There is a role for that: https://ansible.build/#roles

  • Install and configure the OS
  • Manage firewalls and access rights
  • Deploy Docker containers
  • Manage backups and cron jobs
  • Build Wireguard networks

Hosting

We provide a managed server service.

Every customer has a server. No shared environments.

Every application is a hosting offer.

Scripts

For things that cannot be automated there are scripts: https://ansible.build/scripts.html

Ansible roles rely on these helper scripts. Helper scripts work independent of Ansible.

Monitoring

To monitor servers, containers and applications we use Prometheus/Grafana.

Prometheus provides "Exporters" for many applications. Data collection and visualization is simple.

Backup

Backups are defined in Ansible and are done using restic.

  • In Ansible inventory a backup set is defined
  • Ansible creates a cron job to run the backup
  • The backup job uses helper scripts to create local backups
  • The local backups are snapshoted with restic to the backup server
  • All backup data on the backup server is mirrored to an S3 bucket

Proxy

On every server there is Nginx-instance running.

The Nginx config is generated by Ansible: https://ansible.build/roles/nginx/

Some roles require specific configs: https://ansible.build/roles/collabora_code/#nginx-config

Certificates are managed with certbot and let's encrypt.

Auth

We a running Keycloak and integrate it using OAuth/OpenID Connect.

It is possible to manage all Keycloak config in Ansible.

Security

We try to apply best practices:

  • Update our applications / Docker images
  • Linux server patching
  • Basics: fail2ban, ssh pubkey, named users

Nginx WAF with OWASP has been tried. It was too much effort to train.

Challenges

Configuration drift

One an Ansible role or a config is update we have to apply this change to all hosts. This rarely done.

Proxy config not integrated

The proxy config is not provided by the Ansible role. It would be better if the application provides the config.

Backup verification

We cannot be sure that all backups are working. So far they did.