This is a story that dates back to 2017 from one of my client projects at the very beginning of my solopreneur journey. I hope this enlightens some thoughts into the amazing journey of that modern infrastructure has been embarking.
As a part of the digital transformation journey, one of my clients, A Bank in Indonesia, wanted to launch a Digital Banking Product that would give their customers a 24/7 banking solution.
They required a platform that could seamlessly take the load of several thousands of customers who were going to register across different parts of two major cities in Indonesia.
The objective was to provide an AlwayOn PaaS that was highly available with robust monitoring and alerting system and easy to scale out on demand as the utilisation increases.
The Challenges in Always-On Digital Services:
This was a 100% on-premise setup on legacy hardware and core banking systems.
The biggest challenge was to onboard the team to the concept of PaaS, to be live 24/7.
The entire application(a combination of 15 odd micro-services) infrastructure was manually setup on containerisation platform that was meant to be for POC but some how made it to prod.
This, over time, had changes in the system configuration that were not version controlled.
So making any change would lead to risking the active development cycles and in turn impacting production release.
To top it up the tools being used did not provide enough guidelines to build a Live Platform as a Service.
There were other challenges like :
Lack of centralised monitoring and alerting for platform, tools or services.
Data storage did not have high availability of data and was highly unstable.
Lack of proper H/A, DR and Backups strategy.
So how did we tackle all these to build our AlwaysOn Infra. Here are some initiatives that played key roles.
The Initiatives:
The first step was to stabilise the bottom line where the application was running. We re-designed the entire deployment model for the containerisation platform and built a solution that would spin up a containerisation cluster stretched across two datacenters within 20 minutes.
This included
Automated deployment mechanism for containerisation cluster by use of configuration management tools within a CI pipeline
Configuration management to scale out seamlessly on demand, based on the continuous
A deployment model that includes strategies of blue-green, rolling and canary deployment for platform upgrades.
We enabled High Availability for persistent storage using network file storage with geo-replication across datacenters.
Robust monitoring and alerting reports generated for infrastructure utilisation and services health check.
Ensuring production readiness by conducting end-to-end performance testing on the platform for a week prior to application onboarding.
Once the results were confirmed it was released to production in a day.
Once the compute layer was stabilised, we took on to the other layers which were supporting the continuous integration from dev to prod in order to achieve a highly stable and scalable Toolchain that would form the backend of the Continuous Delivery platform that was being built. In order to achieve this goal we:
Analysed multiple POCs and identified the right failover solution for all the backend tools being used for build and deployment.
Created the guidelines and design to achieve H/A, DR, and backups at all levels
Perform various DR Drills to ensure that the process works just fine.
The Final Result:
A stretch cluster on demand across two data centres in 20 minutes.
Always On Platform as a Service for Digital banking Products
Zero downtime for Application and Service layer
Scale out On demand
Geo replication of Data across DC and DR.
RTO and RPO of 1 hr for all backend tools.
If you like this article, I am sure you will find 10-Factor Infrastructure even more useful. It compiles all these tried and tested methodologies, design patterns & best practices into a complete framework for building secure, scalable and resilient modern infrastructure.
Don’t let your best-selling product suffer due to an unstable, vulnerable & mutable infrastructure.
Thanks & Regards
Kamalika Majumder
Comments