★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★Open research. Global perspectives.
editor@theamericanjournals.comPublisher Home ↗
Engineering and Technology OPEN ACCESS

Architecture of a Hybrid In-Memory and Cold Storage for Historical Financial Data

Andrii Humeniuk
Master Degree in Software Engineering Lead Software Engineer, DASTA Incorporated (“dub”) New York, USA,
DOI Crossref DOI 10.37547/tajet/Volume08Issue01-11
tajet 2026
VOL. 8 / NO. 01 JANUARY
FULL TEXT PP. 78-86
DOWNLOAD PDF

View all articles in this issue ↗

VOLUME 8
ISSUE 01
YEAR 2026
PAGES 78-86
RESEARCH OVERVIEW

Abstract

This paper introduces a novel hybrid storage architecture (HyFDS), designed by the author to address the dual challenge of ultra-low latency trading workloads and cost-efficient archival storage in financial markets. Unlike prior works that optimize individual components, HyFDS integrates Apache Kafka, In-Memory Data Grids, lock-free patterns, and Apache Iceberg into a unified framework, validated against the requirements of high-frequency trading. This work proposes a conceptual model of a hybrid data storage architecture (HyFDS) that addresses this problem through the synthesis of heterogeneous technological approaches. The architecture is based on an event-driven model built on Apache Kafka, which serves as a unified bus for all system events. The “hot” tier, implemented on an In-Memory Data Grid (IMDG) and optimized through the LMAX Disruptor pattern and lock-free data structures, enables transaction processing with sub-millisecond latency. The “cold” tier, based on object storage with the Apache Iceberg tabular format, ensures scalable and cost-effective storage. The study analyzes data migration mechanisms, transactional consistency strategies (2PC, Saga), and disaster recovery plans, forming an integrated framework for designing next-generation financial systems.

RESEARCH TOPICS

Keywords

hybrid storage in-memory cold storage event-driven architecture Apache Kafka

Downloads

Download data is not yet available.
SOURCES & SCHOLARSHIP

References

  1. Ionescu, S.-A., Diaconita, V., & Radu, A.-O. (2025). Engineering Sustainable Data Architectures for Modern Financial Institutions. Electronics, 14(8), 1650. https://doi.org/10.3390/electronics14081650
  2. Song, H., et al. (2024). A survey on hybrid transactional and analytical processing. The VLDB Journal, 33, 1485–1515.
  3. Yuan, Z., et al. (2024). Cost-effective data classification storage through text seasonal features. Future Generation Computer Systems, 158, 472–487.
  4. Ren, J., et al. (2021). A machine learning assisted data placement mechanism for hybrid storage systems. Journal of Systems Architecture, 120. https://doi.org/10.1016/j.sysarc.2021.102295
  5. Chi, Y., Liu, H., Peng, G., Liao, X., & Jin, H. (2022). Transformer: An OS-Supported Reconfigurable Hybrid Memory Architecture. Applied Sciences, 12(24), 12995. https://doi.org/10.3390/app122412995
  6. Mamidi, S. (2025). Next-generation data management: Hybrid approaches with Cassandra and Gemfire in financial services. International Journal of Scientific Research in Computer Science, Engineering and Information Technology, 11(2), 1563–1574.https://doi.org/10.32628/CSEIT25112531
  7. Rabelo Ferreira, F. E. R., & do Nascimento Fidalgo, R. (2024). A Performance Analysis of Hybrid and Columnar Cloud Databases for Efficient Schema Design in Distributed Data Warehouse as a Service. Data, 9(8), 99. https://doi.org/10.3390/data9080099
  8. Technavio. Private and public cloud market in financial services to grow by USD 106.43 billion (2024–2028), driven by big data demand; AI driving market transformation. PR Newswire. Retivered from: https://www.prnewswire.com/news-releases/private-and-public-cloud-market-in-financial-services-to-grow-by-usd-106-43-billion-2024-2028-driven-by-big-data-demand-ai-driving-market-transformation---technavio-302352741.html?utm_source (date of access: 10.07.2025)
  9. Lu, K., Zhao, S., & Wan, J. (2024). Hammer: Towards efficient hot-cold data identification via online learning. arXiv, 1-10.
  10. Chang, J., et al. (2024). Idt: Intelligent data placement for multi-tiered main memory with reinforcement learning. In Proceedings of the 33rd International Symposium on High-Performance Parallel and Distributed Computing, 69-82. https://doi.org/10.1145/3625549.3658659
  11. Singh, B., Martyr, R., Medland, T., Astin, J., Hunter, G., & Nebel, J. C. (2022). Cloud based evaluation of databases for stock market data. Journal of Cloud Computing, 11, 53.
  12. Technavio. Private and public cloud market in the financial services industry 2018–2022 (historical market size). Retivered from: https://www.technavio.com/report/private-and-public-cloud-market-in-the-financial-services-industry-analysis (date of access: 10.07.2025)
  13. Telenik S.T. et al. Development and research of models, methods and technologies of planning, programming and management cloudy IT-infrastructures. Retivered from:https://nrat.ukrintei.ua/en/searchdoc/0216U005227/ (date of access: 10.07.2025)