Central log monitoring solution for Erste Group Bank AG
Real-time data streaming
10,000 logs per second
Implementing real-time data
streaming to meet regulations
and allow fast internal analysis.
The Profinit team helped us to define proper HW requirements and then provided the desired solution with the cutting-edge architecture tailored precisely to our needs.Aleš Nedbálek
Service Owner IT Asset Protection
The Project Background
Česká spořitelna a.s., the Czech arm of the Erste Group Bank AG, was looking for a custom solution for the central monitoring of log data from their operational banking systems. The central log collects and stores a massive amount of data – over a specific period of time – to analyse trends or record events from various banking systems plus network and IT environments.
Improvements to their data collection and storing processes had to be made to comply with the new cyber security regulations. But the bank also wanted their in-house security specialists to be able to analyse events as soon as possible after they are logged.
The Business Needs
The solution needed to meet the following specifications:
- Fulfil requirements of new cyber security legislation
- Fast and stable data processing
- Able to analyse data including freshly logged events
- Scalable solution for future data volume increases
The bank’s central monitoring system was based on batch processing of system logs using a standard relational database. It wasn’t compliant with new legislation as it didn’t collect complete data from every system. Plus, the existing system couldn’t meet the new requirements to retain data for a minimum period of time.
Aside from the compliance issues, this system’s operational mode didn’t allow the use of analytical queries that would include log data from the current date of generation – meaning rapid analysis was impossible.
We developed a system that can process streams of logging data from all monitored banking systems in real-time. Working with the bank’s security, IT and operations departments, we tailored the solution to meet each of their specific needs.
Logging data is streamed from newly created central storage through Apache Kafka into a new, dedicated system. There, the data is being processed using Apache Spark and Spark Streaming for monitoring purposes. The whole solution can handle tens of thousands of log entries per second.
Thanks to the implementation of Apache Spark, using lambda architecture, we ensured a uniform handling of stream and batch processing of data. The same code is shared for both processing methods. There is no need to maintain two separate codebases, which makes development, deployment and servicing much simpler and faster. The processed data is being stored in Hive tables within a Hadoop cluster.
Rapid analysis of security data
The unified data overview provides access for specialists in the security department for their analyses. The data is shown in the same format as it was previously, so no analytical process changes had to be implemented. However, it is now possible to access data within minutes of it being generated.
The Tech Stack
We developed and implemented a new system for fast, compliant data processing, and achieved these results for the bank:
- Česká spořitelna a.s. now has a central log monitoring system with uniform handling of real-time and batch data
- This solution processes tens of thousands of logged entries per second
- The system is compliant with new cyber security legislation
- It enables analyses of freshly generated data for the security department
Would your bank benefit from accessing this cutting-edge technology?
Let us show you how Profinit can improve the way you use and access data within your organisation…
Related success stories and use cases
Česká spořitelna Computing anti-fraud predictors
How Profinit helped the Česká spořitelna (Erste Group) dramatically speed up fraud detection, to process 1.5 billion transactions per day.Learn More
Raiffeisenbank Big Data Hadoop Platform
Profinit delivered an end-to-end big data platform, enabling Raiffeisenbank CZ to perform use case analyses with large volumes of transactional data.Learn More