Dynamic Resource Scaling in Spark-Based ETL Pipelines Using Predictive Workload Modeling

The increasing adoption of cloud-native data platforms has led to widespread use of Apache Spark for large-scale Extract, Transform, and Load (ETL) operations. Spark-based ETL pipelines process massive volumes of structured and unstructured data, enabling organizations to support real-time analytics, business intelligence, and data-driven decision-making. However, the performance of these pipelines is highly dependent on efficient resource allocation. Traditional static provisioning and reactive scaling mechanisms often struggle to handle fluctuating workloads, resulting in resource underutilization, increased operational costs, execution delays, and reduced system efficiency. These challenges become more significant in cloud environments where workload characteristics change dynamically over time. This study examines the role of predictive workload modeling in enabling dynamic resource scaling for Spark-based ETL pipelines. The proposed approach utilizes historical workload patterns, execution metrics, and performance indicators to forecast future resource requirements and proactively adjust computational resources. By integrating predictive analytics with resource management strategies, the framework aims to improve cluster utilization, reduce execution latency, and optimize cloud infrastructure costs. Simulation-based experimental results indicate that predictive scaling reduces average job execution time from 145 minutes to 92 minutes, improves CPU utilization from 58% to 86%, improves memory utilization from 62% to 83%, and reduces monthly cloud cost from $8,400 to $5,980 compared with static resource allocation. The study highlights the potential of predictive workload modeling to enhance scalability, performance, and cost efficiency in modern Spark-based ETL environments while supporting the development of intelligent and autonomous data processing systems.