Machine Learning system forecasting transaction anomalies for a Thai financial-sector organization

How the brief came about

The challenge in this case was to know in advance that transactions in a large-scale financial transaction system were about to become abnormal, rather than finding out once a problem had already occurred. The customer is an organisation in Thailand's financial sector that manages application systems and IT equipment across multiple computer centres. The customer wanted a Machine Learning system that would analyse, predict and alert when transaction volume or transaction response times begin to deviate from normal.

A large-scale financial transaction system has to run continuously. When response times begin to slow, the impact reaches many users at the same time. Advance forecast values give the system administration team time to check which transaction route or bank an anomaly started at, before deciding how to respond.

Requirements and constraints

The system had to forecast from the customer's real transaction data and had to avoid affecting the service of the existing system. The main requirements and constraints were as follows.

• The new system had to work alongside the existing systems without affecting service. The customer set this requirement from the start of the project.
• The ML system runs on the customer's servers, both production servers (Production) and development and test servers (Development).
• Transaction data had to be converted and cleaned before being used to train the model. In the early phase, obtaining complete data and gaining access to it took a great deal of time.
• The project's requirements documents are confidential to the customer.

Why we chose an LSTM model and installed it separately from the transaction exchange system

We chose LSTM because it is a neural network suited to time-ordered data, and transaction response times are data of this kind. We installed the ML system on servers separate from the transaction exchange system because the customer specified that the new system must not affect service. The ML system therefore receives transaction data from the existing system for analysis.

1. 1

   Collect and clean the data

   We take transaction data from the customer's transaction exchange system, convert its format, check its continuity and remove unusable data before passing it on to model training.

2. 2

   Train the LSTM model

   The model learns the patterns of transaction response times, and the system then saves the model files and training reports on the server.

3. 3

   Forecast 5 and 10 minutes ahead

   The model forecasts overall, by transaction route and by bank. We tested the forecast results both retrospectively against historical data (Back-test) and in real time (Real-time).

4. 4

   Display on a dashboard

   Graphs on the Grafana Dashboard compare actual values with forecast values and statistics, so the system administration team can monitor them from a single screen.

5. 5

   Keep the system ready for use

   The model retraining system (Retrain) is used to update the model when new data becomes available. The data backup and restore system, together with automatic archiving of old data, helps prevent server storage from filling up.

What was installed and summarised in the project closure document

The work installed on the customer's Production and Development servers and summarised in the project closure document is as follows.

• Trained LSTM model with a training report
• A system for forecasting transaction response time anomalies 5 and 10 minutes ahead, with both Back-test and Real-time testing
• Grafana Dashboard showing graphs of actual values against forecast values and statistics
• Model retraining system (Retrain)
• Data backup and restore system (Backup & Restore) and automatic archiving of old data
• System design documents and the project closure document

Project stages

The project ran from 2021 to 2023 as follows.

1. 2021: Project start; collection and conversion of transaction data for model training
2. 2022: Project closure document prepared, followed by entry into the maintenance period under the contract
3. 2023: Trial training of new models with CNN-LSTM, Random Forest and XGBoost to compare accuracy at each forecast horizon

Confirmed results from the project closure document

We developed a Machine Learning system that forecasts transaction response time anomalies 5 and 10 minutes ahead for an organisation in Thailand's financial sector, and prepared the project closure document in 2022. This document summarises the model training results, the Back-test and Real-time forecast results, the data backup system and the system design documents. This case has no measured figures that can be published, so we report only what is confirmed by documents.

In 2023 we trialled training new models with 3 other algorithms, namely CNN-LSTM, Random Forest and XGBoost, to compare accuracy at each forecast horizon. The recommendations from the trial were to train the models further for higher accuracy and to consider using server hardware data as an additional input.

What we learned

This project offers 3 principles for forecasting anomalies in IT systems.

• Data readiness determines the schedule more than model selection does. In this project, obtaining complete transaction data and converting it into a usable form was the main work of the early phase, so data access rights should be agreed and samples of real data checked before delivery dates are set.
• Where a system must not affect service, the forecasting component should be kept separate on its own server and receive data from the main system for analysis. A separate server means models can be retrained without touching the main system.
• A forecasting model needs a post-installation maintenance plan. Forecast results must be checked both retrospectively and in real time, and a model retraining system must be in place from the start. Any part whose testing is not finished, such as HA, must have its status clearly stated in the project closure document.

How to apply this approach to your system

If your organisation has time-ordered system operating data and wants early warning signals, you can use the same approach as this case: keep the forecasting system separate on your own server, train the model on real data, and display actual values against forecast values on a dashboard. What needs adapting is the input data, the forecast horizon and the measurement method.

• Have access rights to transaction data and historical data ready from the start, and check the data for completeness and continuity.
• Agree together how accuracy will be measured, both retrospectively and in real time, and how far ahead to forecast.
• Define the scope of post-installation maintenance clearly, for example the cycle for checking forecast results and the cycle for retraining models.
• Consider adding server hardware data as a model input, as recommended by the trial in 2023.