Success story / Central log monitoring for security

Central log monitoring solution for Erste Group bank

Real-time data streaming

10,000 logs per second

Scalable architecture

Project Brief

Implementing real-time data
streaming to meet regulations
and allow fast internal analysis.

The Profinit team helped us to define proper HW requirements and then provided the desired solution with the cutting-edge architecture tailored precisely to our needs.
Aleš Nedbálek
Service Owner IT Asset Protection at Česká spořitelna

Project background

Česká spořitelna, the Czech arm of the Erste Group, was looking for a custom solution for the central monitoring of log data from their operational banking systems. The central log collects and stores a massive amount of data – over a specific period of time – to analyse trends or record events from various banking systems plus network and IT environments.

Improvements to their data collection and storing processes had to be made to comply with the new cyber security regulations. But the bank also wanted their in-house security specialists to be able to analyse events as soon as possible after they are logged.

Business needs

The solution needed to meet the following specifications:

Fulfil requirements of new cyber security legislation
Fast and stable data processing
Able to analyse data including freshly logged events
Scalable solution for future data volume increases

Challenge

The bank’s central monitoring system was based on batch processing of system logs using a standard relational database. It wasn’t compliant with new legislation as it didn’t collect complete data from every system. Plus, the existing system couldn’t meet the new requirements to retain data for a minimum period of time.

Aside from the compliance issues, this system’s operational mode didn’t allow the use of analytical queries that would include log data from the current date of generation – meaning rapid analysis was impossible.

Solution

We developed a system that can process streams of logging data from all monitored banking systems in real time. Working with the bank’s security, IT and operations departments, we tailored the solution to meet each of their specific needs.

Logging data is streamed from newly created central storage through Apache Kafka into a new, dedicated system. There, the data is being processed using Apache Spark and Spark Streaming for monitoring purposes. The whole solution can handle tens of thousands of log entries per second.

Thanks to the implementation of Apache Spark, using lambda architecture, we ensured a uniform handling of stream and batch processing of data. The same code is shared for both processing methods. There is no need to maintain two separate codebases, which makes development, deployment and servicing much simpler and faster. The processed data is stored in Hive tables within a Hadoop cluster.

Rapid analysis of security data

The unified data overview provides access for specialists in the security department for their analyses. The data is shown in the same format as previously, so no analytical process changes had to be implemented. However, it is now possible to access data within minutes of it being generated.

Tech stack

Apache Kafka
Apache Spark
Spark Streaming
Hadoop
Java
Apache Hive
Apache Impala

Project Summary

We developed and implemented a new system for fast, compliant data processing, and achieved these results for the bank:

Česká spořitelna now has a central log monitoring system with uniform handling of real-time and batch data
This solution processes tens of thousands of logged entries per second
The system is compliant with new cyber security legislation
It enables analyses of freshly generated data for the security department

Would your bank benefit from accessing this cutting-edge technology?

Let us show you how Profinit can improve the way you use and access data within your organisation…

TALK TO ONE OF OUR BIG DATA SPECIALISTS