Modern infrastructure faces various challenges especially due to manual setups which leads to untraceable changes in systems making every server a work of art on its own.
Then there are classic issues like code that works in a local machine or container, but fails in Production, testing takes forever etc.
Environment On Demand is a setup of having an environment that can be brought up, tested, decommissioned or recreated on demand within a few minutes. This enables performance based auto scale, continuous delivery, backward compatibility and immutable infrastructure.
Basically you treat environments with commodity items. Of course not Production :) because that will be disastrous.
For secure, scalable and sustainable software development and delivery you need multiple production-like environments on demand so end to end testing can happen faster and efficiently.
So here's what we need, to create Environment On Demand :
Modular Infrastructure As Code:
Don’t just automate, modularise and parameterise server configurations, relationships with other servers in code.
Server configuration, packages installed, relationships with other servers etc should be modelled with code to be automated, removing error prone manual steps.
For example, If you are using terraform don't create one module and try to execute every resource from that. Instead create independent modules for each infra component and a separate module to provision these. These can be called module and provisioners. Here is how modules and provisioners structure for can look like.
Configuration Management:
Automate model server configurations, parameterise everything, relationships with other servers, packages in code. This is called Configuration Management.
Configuration is especially useful for software upgrades, patching, files/folder creation, user management and more. This code should always be version controlled.
For instance we can (and should!)
Tag, branch and release the code that defines our servers.
Have a lifecycle that covers different stages through the infrastructure code, ie. dev, QA, production.
Continuously test your infrastructure as you make changes.
Test Driven Delivery:
Test before delivery. Host a production environment right at your dev laptop and the same in QA. So all engineers(Devs, QAs, BAs etc) have a prod like environment to test the same code that will eventually go into production.This way more bugs are caught before it goes to production.
Write tests to validate functionality of each module. This must run each up the module gets configuration update to ensure the base functionality works
Integration tests are needed to test service to service, app to service, service to db or integration with third party services.
Infrastructure must be performance tested to meet the benchmarks before going live in production.
User acceptance tests to ensure business scenarios must be ensured.
Pipeline As Code:
When it comes to infrastructure provisioning it's better not to just execute commands manually from a local host or any remote machine. Instead create parameterised jobs/tasks that can be executed in stages in a pipeline.
Create version controlled CI/CD pipelines as code for provisioning infrastructure. There must be separate stages for specific tasks. For example, you can have one pipeline for creating a Kubernetes cluster with separate stages for different environments like dev, test, prod etc.
Pipeline as code can be used beyond provisioning, for upgrades, patching, data migration, in short any automated task that happens in infrastructure.
And not to mention the logs and historical tracing reports that can be generated from pipeline code are a must during infrastructure audits. It's not possible to do the same from any cli or terminal of a standalone machine.
Other benefits include parallel execution of a variety of tasks from a CI/CD platform, which otherwise is not possible from a standalone machine.
Zero Downtime Deployment:
A deployment model where zero downtime occurs for the end user services, in short no downtime experienced by the users using that service/platform/system.
This includes strategies for blue-green, rolling and canary deployment for platform upgrades, infrastructure scaling, new deployment or any other change event.
To achieve this high availability for persistent storage using network file storage with geo-replication across data centers is a must.
Centralised Command Center:
One stop centralised command center for infrastructure administration
Prevents vendor lock-in.
Centralised logging, monitoring and identity management for better observability
Summary:
EOD ensures test driven delivery for consistent and reliable releases.
Production like the environment on your machine in minutes will be a reality now ensuring consistency across all environments.
Automated and scalable environments on demand can enable faster time to market by making production releases from weeks to days.
If you like this article, I am sure you will find the 10-Factor Infrastructure even more useful. It compiles all these tried and tested methodologies, design patterns & best practices into a complete framework for building secure, scalable and resilient modern infrastructure.
Don’t let your best-selling product suffer due to an unstable, vulnerable & mutable infrastructure.
Thanks & Regards
Kamalika Majumder
Comments