Infrastructure Mint System
A short introduction into the infrastructure of Mint System.
Drivers
Early on we wanted to be able to self-host web application.
ERP-Systems are operating system for companies
- They have to be hosted reliably
- Companies want know where their data is
Infrastructure as Code:
- Every system / server can be rebuilt from code
- The deployment of configuration must be automated
Run apps with Docker containers:
- Manage containers not apps
- Every decent web app provides Docker images
Hardware
We don't want to manage hardware. We use these providers:
- Hetzner
- Exoscale
- Infomaniak
- Ungleich
And service partners.
Ansible
Everything is done with Ansible. There is a role for that: https://ansible.build/#roles
- Install and configure the OS
- Manage firewalls and access rights
- Deploy Docker containers
- Manage backups and cron jobs
- Build Wireguard networks
Hosting
We provide a managed server service.
Every customer has a server. No shared environments.
Every application is a hosting offer.
Scripts
For things that cannot be automated there are scripts: https://ansible.build/scripts.html
Ansible roles rely on these helper scripts. Helper scripts work independent of Ansible.
Monitoring
To monitor servers, containers and applications we use Prometheus/Grafana.
Prometheus provides "Exporters" for many applications. Data collection and visualization is simple.
Backup
Backups are defined in Ansible and are done using restic.
- In Ansible inventory a backup set is defined
- Ansible creates a cron job to run the backup
- The backup job uses helper scripts to create local backups
- The local backups are snapshoted with restic to the backup server
- All backup data on the backup server is mirrored to an S3 bucket
Proxy
On every server there is Nginx-instance running.
The Nginx config is generated by Ansible: https://ansible.build/roles/nginx/
Some roles require specific configs: https://ansible.build/roles/collabora_code/#nginx-config
Certificates are managed with certbot and let's encrypt.
Auth
We a running Keycloak and integrate it using OAuth/OpenID Connect.
It is possible to manage all Keycloak config in Ansible.
Security
We try to apply best practices:
- Update our applications / Docker images
- Linux server patching
- Basics: fail2ban, ssh pubkey, named users
Nginx WAF with OWASP has been tried. It was too much effort to train.
Challenges
Configuration drift
One an Ansible role or a config is update we have to apply this change to all hosts. This rarely done.
Proxy config not integrated
The proxy config is not provided by the Ansible role. It would be better if the application provides the config.
Backup verification
We cannot be sure that all backups are working. So far they did.