This paper presents a modular and reusable system for processing, modeling, and visualizing weather data, as well as a proof of concept implementation focusing specifically on rainfall forecasting. The system was designed to address the challenges of building and maintaining weather data systems by enabling flexible, scalable, and standardized workflows. The proof of concept utilizes ERA-5 Land, the same dataset used in our previous rainfall forecasting research. The objectives were twofold: to implement a generalized system architecture for weather data systems and to validate this architecture through practical implementation. The proof of concept involved creating a model ensemble using machine learning techniques such as autoencoders and transformers, while calculating a variety of error metrics for model evaluation. Results show promising accuracy in rainfall forecasts, with a mean absolute error of 0.23 mm/hour. While the study does not delve deeply into model optimization, it emphasizes data-driven insights and lays the groundwork for future, more refined weather forecasting systems.

Recommended Citation: Neill E., Ricanek K., Dogan G., Modaresnezhad M., (2024). Scalable Data Systems for Weather Forecasing. UNCW MS CSIS Proceedings. V. 18 , N. 18 .