Evaluation of Data Lake and Apache Spark Technologies for Urban Infrastructure Planning and Management

Abstract

This study evaluates the use of Data Lake technology and Apache Spark in the context of urban infrastructure management. By analyzing their capabilities for handling structured, semi-structured, and unstructured datasets, the research highlights their potential to optimize data processing workflows. The system was deployed on Yandex Cloud, leveraging distributed computing and horizontal scalability to achieve efficient data storage, real-time analytics, and fault tolerance. Automation pipelines and quality assurance mechanisms were implemented to streamline data ingestion, transformation, and validation processes. The findings demonstrate significant improvements in data processing efficiency, scalability, and resource optimization, offering a robust framework for enhancing smart city infrastructure planning and evaluation.