Be it performance, scalability or security, modern infrastructure depends heavily on how its storage layer is configured and maintained. Storage or data layer is the core of modern infrastructure.
In an event of a disaster, any other infra resource can be brought back up with minimal interruption in connectivity, but if any data in the storage layer is lost, it can cost the end of the entire business.
There are various kind of data stored in an infrastructure that can be broadly classified as:
Data at rest: Relational/non-relational user data, secrets, static assets, backups etc.
Data in Transit: Communications, file transfer, key exchange etc.
Both kinds need to be protected from failure of any kind, be it availability, scalability, security or cost effectiveness. And here's how data security, scalability & sustainability can be achieved for modern infrastructure:
Multi Site Data Replication:
For high availability, data must be replicated across multiple sites. This is true especially for data at rest. That is why, I always use managed services or database clusters with multiple availability zone replications. These services also help to achieve zero downtime deployments during major os or version upgrades.
Backups are still necessary for compliance and regulations. Enable automated backup of the data instead of machine snapshots.
Do not host critical databases on standalone compute or local system disks, object storage/NFS/NAS as these are difficult to scale and restore. Application data must be hosted on scalable and highly available database clusters.
Configuration Management:
All storage configurations such as provisioning, scaling, migrations or even decommissioning, must be automated through configuration management and infrastructure as code. No manual intervention should be allowed.
Changes in data configuration or structure must be version controlled inline with application configuration.
This will enable traceability for security audits and ensure backward compatibility of data.
Encryption & Secret Management:
Storage systems must be encrypted for data at rest with a centralised key management system with periodic key rotation.
Data in transit such as inter account/region communications must be segregated and encrypted with SSL/TLS.
You can also enable zero trust policy via Mutual TLS to add an extra layer of security for data communication with third party softwares.
Some businesses still prefer to choose a self managed model for data localisation, privacy and security regulations. They are basically worried about IP security on systems managed by someone else. In such a case, you can actually discuss with the cloud provider and get them to clarify their compliance readiness in an agreement.
FinOps:
Storage expenses constitute the most from the infrastructure budget. So planning for storage requirements must be done in advance or it can overspill your entire infra budget.
For example, your current data size is 500 GB, however your estimated utilisation can reach 1 TB, in self managed services you will be reserving 1TB in advance and then keep paying for the empty 500GB.
That's where managed services are recommended for data storage. They charge for the utilisation and are scalable on demand. As a matter of fact, you can strike a good discount if you forecast storage requirements in advance with your cloud provider.
Managing Data Storage For Modern Infrastructure:
To become best seller you need to establish data trust with your users, by ensuring its availability, accessibility and security. For highly available, performant and secured services, data storage must be designed as follows:
Replication over restoration is the ideal model for ensuring availability & recovery of data either during a disaster or for standard maintenance activities.
Version controlled configuration and data is a must for traceability and backward compatibility.
Databases on compute systems with application must be avoided as they wont allow for zero downtime deployments.
Data at rest must always be on Object Storage(when on cloud) or NFS/NAS when on-premise so that it can be replicated/available real time across multiple sites.
Some security compliances such as banking, financial services etc still ask for regular backups and restoration of data. In such cases, backup the data instead of machines/disks. Full machine backups are prone to disk/data corruption and are costly.
Encryption of data at rest with automated key management is a must for ensuring security of data against external threats and leaks.
If you like this article, I am sure you will find 10-Factor Infrastructure even more useful. It compiles all these tried and tested methodologies, design patterns & best practices into a complete framework for building secure, scalable and resilient modern infrastructure.
Don’t let your best-selling product suffer due to an unstable, vulnerable & mutable infrastructure.
Thanks & Regards
Kamalika Majumder
Commentaires