INTELLIGENT WORKLOAD PLACEMENT AND SHARDING BETWEEN ORACLE, POSTGRESQL, MONGODB, AND CASSANDRA USING MACHINE LEARNING

Authors: Adithya Sirimalla

DOI: https://doi.org/10.37082/IJIRMPS.v9.i3.232851

Short DOI: https://doi.org/hbd6hw

Country: United States

Full-text Research PDF File:   View   |   Download

Abstract: Polyglot persistence has become a popular trend in the modern world, and it is now common to find modern applications that depend on heterogeneous datastores, including Oracle, PostgreSQL, MongoDB, and Cassandra. Both engines have unique benefits in terms of consistency, latency, scalability, and data modeling; however, neither one of them is the best system to use for each workload pattern. Static location heuristics do not reflect real-life workload variations, which results in a reduction in performance, unwarranted resource expenses, and complexity of operations. In the current study, there is an Intelligent Workload Placement and Sharding Advisor that is an ML-based tool that examines workload signatures (read/write ratios, latency percentiles, payload distributions, access skew, and transactional requirements) to advise the best setting of data placement among heterogeneous engines. It has a structure that combines the use of supervised machine learning to predict latency and throughput, an engine recommender that uses classification, and a reinforcement learning migration policy that compares the improvement in long-term performance with migration overhead. Synthetic OLTP traces using YCSB workloads, mixed read/write patterns, and synthetic OLTP traces show that the proposed system significantly improves over the static baselines, such as in reducing p95 latency, maximum throughput at bursty workloads, and SLA violations. The study shows that automated workload-aware decision systems can be used to enhance performance and minimize the operational load in multi-engine database systems.

Keywords: Workload-aware placement, Sharding, Oracle, PostgreSQL, MongoDB, Cassandra, Machine learning for databases, reinforcement learning, distributed storage, and YCSB benchmarking.

Paper Id: 232851

Published On: 2021-05-07

Published In: Volume 9, Issue 3, May-June 2021

Share this