Data Engineer - Azure Databricks (5-10 yrs)

Key Responsibilities :

- Framework Development & Optimization : Design and develop modular, reusable, and scalable Python-based frameworks to support data engineering and processing tasks.

- Data Engineering & Transformation : Implement and maintain frameworks that facilitate the ingestion, transformation, and processing of large datasets using Azure services.

- Pipeline Orchestration & Automation : Develop and optimize data engineering pipelines leveraging Azure Function Apps, Databricks (PySpark), and Apache Airflow to ensure efficient data processing.

- Real-Time & Event-Driven Architectures : Design and implement real-time data streaming solutions and event-driven architectures using Azure Event Hub.

- Data Warehousing & Dimensional Modeling : Work with dimensional modeling concepts to build robust data warehouse architectures, ensuring optimized storage and retrieval for analytics and reporting.

- Integration with Azure Data Services : Architect and integrate frameworks with Azure Data Lake, Azure Storage, and Cosmos DB to enable seamless data flow and management.

- Serverless Data Engineering Solutions : Design and deploy serverless solutions using Azure Function Apps to support scalable and cost-efficient data engineering frameworks.

- CI/CD & DevOps Implementation : Develop and manage CI/CD pipelines using Azure DevOps, ensuring streamlined integration, testing, and deployment of data engineering frameworks.

- Automated Testing & Monitoring : Implement automated testing, deployment, and monitoring processes to maintain high-quality code, performance optimization, and rapid iteration.

Required Skills & Experience :

- Strong expertise in Python, with a focus on developing scalable frameworks for data engineering.

- Hands-on experience with Azure cloud services, including Azure Function Apps, Event Hub, Data Lake, Storage, and Cosmos DB.

- Proficiency in PySpark and Databricks, with experience in handling large-scale data processing and transformation.

- Strong background in Apache Airflow for workflow orchestration and pipeline automation.

- Expertise in Data Warehousing concepts, including dimensional modeling, star schema, and optimized data storage.

- Experience in real-time data streaming and event-driven architectures using Azure Event Hub.

- Strong understanding of CI/CD principles with hands-on experience in Azure DevOps for deployment automation.

- Knowledge of automated testing, monitoring, and logging for maintaining high-performance data pipelines.

- Problem-solving mindset with a focus on scalability, efficiency, and automation in data engineering solutions.