Engineering and Technology | Open Access | DOI: https://doi.org/10.37547/tajet/Volume08Issue02-02

Real-Time Log Analytics in Distributed Systems: Minimal-Latency Detection of Critical Events for Cloud-Native Back-End Platforms

Ivan Akimov , Software Engineer Dubai, United Arab Emirates

Abstract

The paper examines real-time log analytics for distributed, cloud-native back-end systems, where operational decisions depend on the rapid recognition of critical runtime conditions. The relevance follows from the latency sensitivity of microservice-based finance and trading workloads, where propagation of failures, retries, and cascading timeouts rapidly degrades user-facing and internal processing. The novelty lies in an integrated analytical synthesis that ties stream-processing scalability evidence, tracing-tool capabilities, monitoring-tool taxonomies, instrumentation overhead studies, and modern log-anomaly detection research into one consistent engineering narrative. The study aims to develop a low-latency detection approach based on peer-reviewed findings. To achieve this goal, the work employs a systematic selection of recent literature, structured extraction of architectural patterns, and comparative reasoning across the ingestion, correlation, detection, and alerting stages. The analysis encompasses distributed stream processing benchmarks, near-real-time processing in practical architectures, runtime verification for streaming systems, and state-of-the-art log anomaly detection methods. The closing part derives design implications for practitioners building observability and incident-response pipelines.

Keywords

real-time log analytics, distributed systems, stream processing, observability, microservices, critical event detection, low latency, tracing, anomaly detection, cloud platforms

References

Aladib, L., Su, G., & Yang, J. (2025). Real-Time Monitoring for Distributed Stream Processing Systems Using Linear Temporal Logic. Electronics, 14(7), 1448. https://doi.org/10.3390/electronics14071448

Daksa, D., & Kemala, E. (2025). Comparative Analysis of Apache Flink and Apache Spark for Near Real-Time Fraud Detection in Stream Processing Architecture. Procedia Computer Science, 242, 4691–4698. https://doi.org/10.1016/j.procs.2024.11.247

Duan, X., Du, D., Liu, Z., Zhu, H., & Liang, C. (2024). LogEDL: Log Anomaly Detection by Evidential Deep Learning. Applied Sciences, 14(16), 7055. https://doi.org/10.3390/app14167055

Giamattei, L., Guerriero, A., Pietrantuono, R., Russo, S., Malavolta, I., Islam, T., … Simon Panojo, F. (2024). Monitoring tools for DevOps and microservices: A systematic grey literature review. Journal of Systems and Software, 208, 111906. https://doi.org/10.1016/j.jss.2023.111906

Hammad, M., Ahmad, A. & Andras, P. (2025). An empirical study on the performance overhead of code instrumentation in containerised microservices. Journal of Systems and Software, 230, 112573. https://doi.org/10.1016/j.jss.2025.112573.

Henning, S., & Hasselbring, W. (2024). Benchmarking scalability of stream processing frameworks deployed as microservices in the cloud. Journal of Systems and Software, 208, 111879. https://doi.org/10.1016/j.jss.2023.111879

Janes, A., Li, X., & Lenarduzzi, V. (2023). Open tracing tools: Overview and critical comparison. Journal of Systems and Software, 204, 111793. https://doi.org/10.1016/j.jss.2023.111793

Khan, Z. A., Shin, D., Bianculli, D., & Briand, L. C. (2024). The Impact of Log Parsing on Deep Learning-based Anomaly Detection in System Logs. Empirical Software Engineering, 29, 10. https://doi.org/10.1007/s10664-023-10440-5

Landauer, M., & Skopik, F. (2023). Deep learning for anomaly detection in log data: A survey. Internet of Things and Cyber-Physical Systems, 5, 100071. https://doi.org/10.1016/j.iotcps.2023.100071

Li, W., Wu, Y., Huang, W., Ou, W., Wang, H., Zhou, F., & Deng, L. (2025). System log anomaly detection based on contrastive learning and retrieval augmented. Scientific Reports, 15, 38370. https://doi.org/10.1038/s41598-025-22208-7

Download and View Statistics

Views: 0   |   Downloads: 0

Copyright License

Download Citations

How to Cite

Akimov, I. (2026). Real-Time Log Analytics in Distributed Systems: Minimal-Latency Detection of Critical Events for Cloud-Native Back-End Platforms. The American Journal of Engineering and Technology, 8(2), 08–16. https://doi.org/10.37547/tajet/Volume08Issue02-02