Data Engineer - ETL/Azure (5-10 yrs)

Job Post : Sr. Data Engineer (Azure).

Experience : 5+ years.

Location : Ahmedabad (Hybrid Model).

Responsibilities :

- Design, develop, and maintain scalable and efficient data pipelines using industry standard tools and technologies.

- Implement and optimize ETL processes to ensure data quality, reliability, and performance.

- Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver robust data solutions.

- Utilize Apache Spark for largescale data processing and analysis.

- Manage containerized applications using Docker and orchestrate them with Kubernetes.

- Automate workflows and data pipelines using Apache Airflow.

- Leverage Azure cloud services for data storage, processing, and analytics.

- Design and maintain SQL Server databases, ensuring high availability and performance.

- Implement data quality checks using Great Expectations to ensure data accuracy and consistency.

- Contribute to the overall system design and architecture, ensuring scalability and maintainability.

- Troubleshoot and resolve data related issues, ensuring minimal downtime and data loss.

- Stay current with emerging trends and technologies in data engineering and propose new solutions to enhance the data infrastructure.

Requirements :

- 5+ years of experience in data engineering or a related field.

- Proficient in Python for data manipulation and automation.

- Hands-on experience with Apache Spark for big data processing.

- Extensive experience with Docker for containerization and Kubernetes for orchestration.

- Proficient in using Apache Airflow for workflow automation.

- Strong experience with Azure cloud services (i.e., Azure Data Lake, Azure SQL Database, Azure Databricks).

- Expertise in SQL Server databases, including design, optimization, and management.

- Familiarity with data quality tools like Great Expectations.

- Solid understanding of system design principles and best practices.

- Excellent problem-solving skills and attention to detail.

- Strong communication skills and the ability to work collaboratively in a team environment.