SIEM(Security Information & Event Management) On Cloud

May 1, 20244 min read

Updated: May 30, 2024

Did you know logging and SLA are also part of security compliance of an organisation ?

The faster you can debug & solve an issue the better will be your service SLA.

Customer sales increases with the increase of 9s in your SLA.

For efficient and faster debugging you need:

👉 Logging enabled for all infrastructure resources and software components.

👉 Standard log collector with human readable well-formatted logs.

👉 Periodic log rotation, archival and retention policy for optimized and cost effective. Storage.

👉 Server, appliance and system logs, API and console logs, database logs stored in human readable formats.

For faster debugging you need a one stop station of human readable formatted logs, in short a centralised robust logging system that is easy to operate, stable, scalable, secure and cost effective.

This one stop station for traceability is known as the Centralised Logging System, that most software organisations use for application debugging and support.

How to log for security?

One of the mandatory requirements for most global security compliances is having a system called SIEM on cloud.

SIEM stands for Security information and event management. It's basically a security solution that collects all audit level information in infrastructure and applications that helps organisations detect threats before they disrupt business.

These are debug or audit level logs like VPC flow logs, system secure logs, api access logs, control panel, firewall change logs etc
They must be set with a longer log retention policy as per the compliance needs. Sometimes it can be equal to one or multiple years.
Most regulators especially in the banking or financial sector may need you to retail logs for at least a year or two.
The idea is early detection and mitigation of security threats using the logs generated.

Now here comes a million dollar question, which if not answered at the right time can actually cost you if not a million, at a few thousand dollars each month.

Can Centralised Logging & SIEM systems be the same?

Technically speaking, yes you can have a single system to log both the application logs and security logs. And thus we can say that centralised logging and SIEM on cloud can be the on the same platform.

But the real (million dollar one) question is:

Should Centralised Logging & SIEM systems be the same?

Most regulators especially in the banking or financial sector may need you to retail logs for at least a year or two. In such a case logs must be archived and kept in a separate storage. Else you might hit a disk space issue for your log aggregator. And here comes the answer:

If you keep both centralised logging (applications and infrastructure logs ) and SIEM on the same system lets say a Splunk or a Opensearch cluster, here are the challenges you might face:

First and foremost, access control can be a challenge if both are on the same systems. Most organisations which have to comply with regulatory audits refer to separate systems just so that they can prove dedicated SIEM setup. It's easier to convince the auditor.
If not then very granular role based access policies must be defined for users since these can contain PII data which must not be visible to all users.
Cost incurred for storage of SIEM logs which as mentioned above are more than those in normal logging.
The nature of SIEM logs is of debug or audit level and hence more raw data will be collected and hence more active storage.
And lastly centralised logging is essential from Dev to Prod, as it helps debugging app issues during development and testing. Whereas SIEM is a requirement on Production as it is about the actual business security information and event management.

Recently I faced all these questions while building SIEM for my own project.

And here’s what I decided to do:

SIEM On Cloud:

Security is already dreaded by most organisations as the next black hole. So it's better to simplify the setup and avoid conflict between different sections. So rather than putting all eggs in the same basket and applying too many roles and responsibilities on the same system, it's in everyone’s best interest to keep Centralised Logging and SIEM systems separate.
We can always use the same tools/software/platform for both, but keeping two separate instances. You can tune the size accordingly ex: smaller instances for centralised logging with log rotation enabled. Whereas SIEM can be only for prod.
Nowadays there are also platforms and SaaS options available just for SIEM, some of which are also certified by security compliances. These have become very popular among larger organisations which have specific security needs.

To summarise:

Logs are for observability and traceability, not just for checking errors.
Most security requirements are actually just extended levels of daily events.
So don’t just log, log for security.

If you like this article, I am sure you will find 10-Factor Infrastructure even more useful. It compiles all these tried and tested methodologies, design patterns & best practices into a complete framework for building secure, scalable and resilient modern infrastructure.