Reading time: 12-19 minutes – by Henrik L. Mohr in Digital Sovereignty
Librelab has helped a small Danish company, Meewee, move the hosting of their product from Amazon AWS to European servers.
The results are:
- 100% freedom to operate their product wherever they want
- A saving of five-sixths – 83%!
- More technical resources available than before
The following is a description of how we approached the task in general.
The description is provided in agreement with Meewee, and you are welcome to use it as inspiration for how you can approach a similar task in your organisation.
You are also welcome to contact us if you are interested in our suggestions for a strategy for migrating your web-based product to your own servers or servers owned by European providers.
Analysis is always a good place to start:
Step 1: Identify bindings
We started by analysing the product’s close ties to AWS. Some of them are difficult to break, others are easy, and some can wait.
Below are the AWS-specific components that were used. Click on the headings or the plus sign to the right of a heading if you are interested in a little more detail.
Meewee used DynamoDB, which is a so-called NoSQL database, used in this case as a key-value database. DynamoDB is inexpensive as long as the data volumes are not enormous, and at the same time fast and stable.
The product already used what are known as Docker containers, which are a way of packaging software solutions so that they are easier to move from one cloud provider to another, or even to your own servers. However, the product specifically used the AWS variant called ECS Fargate, and the configuration describing how the product should run and scale was specific to this variant.
ECR was used to store the templates on which the running containers were to be based. Read about how we replaced ECR in Step 7: Deployment.
Various tasks ran at different times throughout the day. These included sending emails, but also other types of clean-up tasks. Lambda was the AWS service used for this purpose.
In front of the running applications, the product used a so-called Application Load Balancer, which is also AWS-specific. The SSL/TLS certificates that were in front of the product’s services were configured in this Load Balancer. The certificates were generated by AWS ACM, which issues and renews these certificates.
All application logging and sending of alerts via email was done from CloudWatch, which collects all application logs in a visual dashboard that provides both an overview and details.
Al brugerstyring, det vil sige adgang ikke til selve produktet, men til at administrere det i forskellige niveauer baseret på roller, blev foretaget ved hjælp af AWS IAM.
AWS SES blev benyttet til udsendelse af e-mails. For at få dette til at fungerere korrekt, krævedes specifik konfiguration af forskellige værdier i produktets DNS (Domain Name Service).
Since Meewee is a web-based system that is primarily accessed via a web browser, it is advantageous to use a so-called CDN (Content Delivery Network), which makes the static files (primarily HTML, CSS and JavaScript) available as physically close to the user as possible. Meewee used CloudFront to make the static files available worldwide.
The primary dependency that was not AWS-specific was Github, which was used for storage, collaboration and versioning of source code, but also to ensure that the product was tested and built correctly after changes and moved to the production environment via Github Actions.
Step 2: Identify alternatives
In order to quickly move Meewee’s product out of AWS, the initial task was to make the move with as few changes to the product’s code as possible.
We naturally took up the challenge and ended up being able to move Meewee to a European provider solely by changing various configuration files – in other words, without making any changes to the product’s source code itself.
The headings below show what Meewee has migrated from and to, and again, you can click on the heading for more details.
At Librelab, we have many years of positive experience with the German provider Hetzner, which cannot be compared to either AWS or Azure in terms of features, but which offers exactly what you need if you want to host a solution using Open Source components. Hetzner offers simple add-ons such as load balancers and firewalls, and since these are often the most difficult to configure and set up yourself, it was a perfect fit for this task.
We could also have chosen Danish providers, but since we were in a hurry and already knew Hetzner, it became our recommendation and Meewee’s final choice. Hetzner is also one of the cheapest providers, so we expected to be able to achieve significant savings for Meewee.
The entire installation is backed up once an hour, allowing the entire system to be restored. Data is also backed up independently.
Should Meewee later choose a Danish provider, their product is now, unlike before, easy to migrate.
NB: At the time of writing, Librelab has no commercial or other form of cooperation agreement with Hetzner!
It quickly became apparent that what tied Meewee most closely to AWS was the use of the AWS DynamoDB database, which is fast, inexpensive and incredibly stable. After some research, ScyllaDB was chosen as an alternative. It is an open source database with similar speed and stability, and with a few simple steps, it can be made fully compatible with DynamoDB.
The only disadvantage of ScyllaDB compared to DynamoDB is that you have to set up data backup yourself and ensure that recovery works as it should. Data backup and recovery are done via simple shell scripts.
Meewee was already using Docker containers to run the product under ECS Fargate. Instead of ECS configuration files, we therefore chose to create a Docker Compose file that described the individual services, linked them together, and made them available to the outside world.
We decided not to find an alternative to AWS ECR, as it was not actually necessary in Meewee’s case. More on this later.
When we chose to use Linux as the underlying operating system for the running services, we automatically had cron available. Cron is an incredibly simple and stable system for running tasks at fixed intervals. Hetzner also offers cron at a more general level than the individual services. A combination of these two levels of cron could easily replace Meewee’s previous use of Lambda.
When you have a web-based product, you cannot avoid the need for a load balancer. A load balancer distributes user requests across the available running services and ensures that only services that are ‘healthy’ are called. If a service is ‘unwell’ because, for example, it is running on a physical machine that is defective, due to network problems, or because it is restarting after an update, a load balancer ensures that user requests are only forwarded to the ‘healthy’ services.
Furthermore, SSL/TLS certificates are usually configured in these load balancers. These certificates ensure an encrypted connection between the server and the user’s web browser.
Hetzner offers all these features for a modest additional cost. Load balancers are often quite expensive components, as they are very critical. In Meewee’s case, almost 20% of the total hosting price goes to the load balancer, but this also includes the issuance and renewal of certificates.
The need to examine log files is generally very low in Meewee. It is a very stable product that has been in production since 2012 and has had no crashes in AWS since 2016.
CloudWatch’s visual tools are therefore not needed in the short term. Simple text-based monitoring is fine, and emails sent directly from the server to selected recipients when server errors occur are sufficient. The technicians who need to look at and possibly read an error can subsequently find the relevant log files from the running service.
The potentially very complex configurations of roles and user groups that can be set up in AWS IAM are not at all necessary for the small team behind Meewee.
Linux offers plenty of options for granting access to specific resources and file systems. Only a very limited number of people already have access to the running environments in Meewee’s product.
Emails sent from Meewee are still sent via AWS SES. This is a component that is relatively easy to replace, but there was no time to replace it initially.
The fact that SES has not yet been replaced is not a major problem. Firstly, very few emails are sent from the Meewee application. One example is when a user wants to change their password. Secondly, these emails do not contain critical user information.
Nevertheless, it is a component that will be replaced soon. When the time comes, we have recommended that Meewee use Hetzner’s mail service to consolidate as much as possible in one place.
We have found several good, fast and budget-friendly European alternatives to AWS CloudFront. We have chosen to highlight Bunny CDN, as it gets good reviews, is fast, easy to get started with and reasonably priced.
Step 3: Test locally
First, we created a local version of Meewee that ran on ScyllaDB with a small test database and individually running docker containers (without docker compose).
When it worked, we combined these services with docker compose and tested that it still worked locally.
Step 4: Github to own Gitea
We recommended that Meewee switch from Github to its own Gitea installation. Many who host something similar to Github themselves choose to use Gitlab. However, in our experience, Gitlab is quite resource-intensive compared to Gitea, and Meewee’s need for the extra features Gitlab offers is currently non-existent.
The source code itself is easy to move out of Github and into similar systems. The deployment pipelines Meewee had in Github were very extensive and complex, which was why most of them were disabled or never used.
Gitea has similar ‘Actions’ to Github and covers Meewee’s needs nicely.
Step 5: Package
The Meewee product was initially built using the new docker-compose-based setup and a simple test database, following a manual process.
One of the advantages of using non-proprietary components is that you can easily test the product on your own computers before testing it on a supplier’s servers.
After a final round of configuration changes, the entire product with the new components worked locally.
Step 6: Test with provider
The same package that was tested locally was subsequently copied to a test environment at Hetzner and made available via temporary IP addresses and DNS names.
When this also worked as intended, it was time to define how the deployment of changes would be handled in the future.
Step 7: Deployment
Previously, a complex system of Github Actions was used to move changes in Meewee’s product from Github to AWS. This has now been replaced by a simple and pragmatic system:
- Changes are first tested locally by the developers
- When the developers are ready with their changes, they are sent to Gitea, which uses Gitea Actions to test the entire code base, including the new changes.
- Subsequently, integration tests are performed, which, among other things, start a temporary local environment and perform so-called ‘smoke tests’ that test, among other things, that a user can log in.
- If everything works as intended, the new changes are merged into the source code’s main branch.
- A cron job at Hetzner listens for changes in this main branch every 5 minutes. If there are changes, they are retrieved and a new version of the service in question is built on the server (replacing the need for ECR).
- The use of watchtower in docker compose subsequently captures the changes and restarts the service in question when there are changes.
This simple setup means that there may be downtime for the service being updated. In practice, this has very little impact, as it takes only a few seconds to restart Meewee’s services through Docker.
Although it is a minor issue, it is one that is worth being aware of – some services take longer to restart than others, and during periods with many users, even short periods of downtime are not practical.
One solution could be to update the cron job to only check for changes if there is low load on the service in question. A better solution would be to ensure that at least one service is always running on the old version until the new service is up and running, and then update the old services afterwards (known as blue-green deployment). You can also choose the ‘more expensive’ model – Kubernetes – or the slightly simpler Docker Swarm.
We have recommended that Meewee first look at the server load before updating. And later look at Kubernetes.
Step 8: AWS in maintenance-mode
We put the running application into maintenance mode (also called read-only mode). This meant that users could log into Meewee, but could not create or update data.
This was necessary in order to stabilise the database.
Step 9: New system into operation
The new production environment was started with an empty ScyllaDB database, ready to receive a copy of the data.
Step 10: Backup & Restore
Data was copied from DynamoDB to the new running ScyllaDB production database.
The new production environment was then restarted, and it was checked that it was ready to receive user activity.
Furthermore, it was ensured that there was automatic backup of all new services and databases.
Step 11: Move DNS
Meewee’s DNS was now updated to point to the new running product at Hetzner, while the previous production environment at AWS remained in maintenance mode.
Step 12: Monitor closely
Over the next 24 hours, we worked closely with Meewee to monitor the new production environment and were ready to make changes to the configuration if necessary.
In practice, it took well under 24 hours to migrate active users from AWS to Hetzner, as we had modified the relevant DNS records a few days earlier to expire after 10 minutes. The result was that the vast majority of DNS servers around the world were correctly updated to the new production environment in less than 10 minutes. In layman’s terms, this means that most users logged into the new production environment instead of the old one within 10 minutes of the move.
⏭ Next steps
CDN and E-mail
Meewee still uses AWS CloudFront for CDN and AWS SES for email delivery. As mentioned earlier, Meewee will switch from CloudFront to Bunny CDN and from SES to Hetzner mail service in the near future.
The current cost of the two outstanding services in AWS is 1/180 of the previous total amount, which is an acceptably low level. But most importantly, neither service performs data processing or storage.
Docker to Podman
Podman is often mentioned as the better open source alternative to Docker. We tested Meewee’s new setup with Podman, but encountered some errors we hadn’t seen before, and due to time constraints, we had to abandon the use of Podman for the time being.
Meewee’s wish list includes revisiting this issue as soon as possible and finding a solution.
Kubernetes
To make scaling easier and ensure uptime, even when some of the heavier services are being updated, we have recommended that Meewee look into Kubernetes (K8S).
Kubernetes uses Docker, and it is relatively straightforward to migrate the current setup to K8S.
Operations Dashboard
The last thing Meewee wants in the new environment is a dashboard that gives them a complete real-time overview of their production environment, for example:
- Number of active users
- Load on services (memory, processing power, disk space, network)
- Potential security attacks
- Outdated components in operation
- Underlying operating systems
- Components used by the product code
- Access to log files
Our recommendation is to base this dashboard on Elastic’s open source systems, such as Kibana, which is relatively easy to host on your own environments.
🥇Result: Freedom, savings & better performance
Meewee now has the freedom to run its SaaS product wherever it wants, without specific ties to specific providers.
After using the new production environment for four weeks, it is clear that the savings in pounds sterling are significant. For every €1000 previously paid to AWS, €170 is now paid to Hetzner:
That is a massive saving of 83%!
Add to that the fact that Meewee now has more resources at its disposal, in the form of more RAM, more CPU power and more disk space, and that their knowledge building going forward will be directed towards open solutions rather than vendor-specific solutions, which is something they can benefit from regardless of which vendors they may choose in the future.
How long did the migration take? The preparation described in steps 1 to 7 was completed within a few weeks, and the actual technical migration described in steps 8 to 11 was completed in less than 2 hours.
The result shows a clear and meaningful business case on all parameters.
You and your organisation?
If you would like to know how Librelab can help you become digitally sovereign with your existing or future digital products, please send us an email at 📨 hej@librelab.eu.
We would be happy to invite you for a no-obligation coffee and chat about the possibilities!
(this article has been translated from the original Danish version with the help of Deepl).
Remember that you can subscribe to new posts from our blog!
The newsletter (in Danish) is published every Friday morning and summarises the news from the past week, so you never miss out on relevant tips, tricks, news and articles from Librelab:
📰 Sign up for our newsletter 📰.
Read another similar inspiring story on Pernille Tranberg’s blog, dataethics.eu.
NB: The leadership of Librelab and Meewee has overlaps, which we don’t believe changes the relevance or validity of this article.