This paper introduces a novel hybrid storage architecture (HyFDS), designed by the author to address the dual challenge of ultra-low latency trading workloads and cost-efficient archival storage in financial markets. Unlike prior works that optimize individual components, HyFDS integrates Apache Kafka, In-Memory Data Grids, lock-free patterns, and Apache Iceberg into a unified framework, validated against the requirements of high-frequency trading. This work proposes a conceptual model of a hybrid data storage architecture (HyFDS) that addresses this problem through the synthesis of heterogeneous technological approaches. The architecture is based on an event-driven model built on Apache Kafka, which serves as a unified bus for all system events. The “hot” tier, implemented on an In-Memory Data Grid (IMDG) and optimized through the LMAX Disruptor pattern and lock-free data structures, enables transaction processing with sub-millisecond latency. The “cold” tier, based on object storage with the Apache Iceberg tabular format, ensures scalable and cost-effective storage. The study analyzes data migration mechanisms, transactional consistency strategies (2PC, Saga), and disaster recovery plans, forming an integrated framework for designing next-generation financial systems.

hybrid storage, in-memory, cold storage, event-driven architecture, Apache Kafka

Ionescu, S.-A., Diaconita, V., & Radu, A.-O. (2025). Engineering Sustainable Data Architectures for Modern Financial Institutions. Electronics, 14(8), 1650. https://doi.org/10.3390/electronics14081650

Song, H., et al. (2024). A survey on hybrid transactional and analytical processing. The VLDB Journal, 33, 1485–1515.

Yuan, Z., et al. (2024). Cost-effective data classification storage through text seasonal features. Future Generation Computer Systems, 158, 472–487.

Ren, J., et al. (2021). A machine learning assisted data placement mechanism for hybrid storage systems. Journal of Systems Architecture, 120. https://doi.org/10.1016/j.sysarc.2021.102295

Chi, Y., Liu, H., Peng, G., Liao, X., & Jin, H. (2022). Transformer: An OS-Supported Reconfigurable Hybrid Memory Architecture. Applied Sciences, 12(24), 12995. https://doi.org/10.3390/app122412995

Mamidi, S. (2025). Next-generation data management: Hybrid approaches with Cassandra and Gemfire in financial services. International Journal of Scientific Research in Computer Science, Engineering and Information Technology, 11(2), 1563–1574.https://doi.org/10.32628/CSEIT25112531

Rabelo Ferreira, F. E. R., & do Nascimento Fidalgo, R. (2024). A Performance Analysis of Hybrid and Columnar Cloud Databases for Efficient Schema Design in Distributed Data Warehouse as a Service. Data, 9(8), 99. https://doi.org/10.3390/data9080099

Technavio. Private and public cloud market in financial services to grow by USD 106.43 billion (2024–2028), driven by big data demand; AI driving market transformation. PR Newswire. Retivered from: https://www.prnewswire.com/news-releases/private-and-public-cloud-market-in-financial-services-to-grow-by-usd-106-43-billion-2024-2028-driven-by-big-data-demand-ai-driving-market-transformation---technavio-302352741.html?utm_source (date of access: 10.07.2025)

Lu, K., Zhao, S., & Wan, J. (2024). Hammer: Towards efficient hot-cold data identification via online learning. arXiv, 1-10.

Chang, J., et al. (2024). Idt: Intelligent data placement for multi-tiered main memory with reinforcement learning. In Proceedings of the 33rd International Symposium on High-Performance Parallel and Distributed Computing, 69-82. https://doi.org/10.1145/3625549.3658659

Singh, B., Martyr, R., Medland, T., Astin, J., Hunter, G., & Nebel, J. C. (2022). Cloud based evaluation of databases for stock market data. Journal of Cloud Computing, 11, 53.

Technavio. Private and public cloud market in the financial services industry 2018–2022 (historical market size). Retivered from: https://www.technavio.com/report/private-and-public-cloud-market-in-the-financial-services-industry-analysis (date of access: 10.07.2025)

Telenik S.T. et al. Development and research of models, methods and technologies of planning, programming and management cloudy IT-infrastructures. Retivered from:https://nrat.ukrintei.ua/en/searchdoc/0216U005227/ (date of access: 10.07.2025)