Data Engineer - Python/ETL (6-12 yrs)

Job Description :

We are looking for an experienced Data Engineer with expertise in modern data engineering tools and platforms. The ideal candidate will have strong proficiency in AWS, Azure, Databricks, Apache Airflow, PySpark, and Python, with a proven track record of building and maintaining scalable data pipelines and solutions. This role requires hands-on experience with cloud platforms and a deep understanding of data processing, orchestration, and analytics.

Key Responsibilities :

1. Data Pipeline Development :

- Design, build, and maintain scalable and robust data pipelines using tools like Databricks, Airflow, and PySpark.

- Implement ETL/ELT processes to ensure seamless data ingestion, transformation, and delivery.

- Integrate data from diverse sources into cloud-based data lakes and warehouses.

2. Cloud Data Solutions :

- Leverage AWS and Azure cloud platforms to design and deploy data solutions.

- Optimize data storage and processing on cloud-native platforms for performance and cost efficiency.

3. Data Analytics and Processing :

- Implement batch and real-time data processing pipelines.

- Enable advanced analytics and machine learning by preparing high-quality datasets.

4. Monitoring and Optimization :

- Monitor data pipelines and workflows to ensure reliability and performance.

- Debug and resolve issues, ensuring data accuracy and consistency.

- Continuously optimize workflows for scalability and efficiency.

5. Collaboration :

- Work closely with data scientists, analysts, and other stakeholders to understand requirements.

- Translate business needs into technical solutions and deliver actionable insights.

Technical Skills and Qualifications

Core Technologies :

- Expertise in Databricks, PySpark, and Python for data engineering tasks.

- Proficiency in workflow orchestration tools like Apache Airflow.

Cloud Platforms :

- Hands-on experience with AWS (e.g., S3, Redshift, Glue, EMR).

- Experience with Azure (e.g., Azure Data Factory, Synapse Analytics).

Data Management :

- Strong understanding of data warehousing concepts and best practices.

- Knowledge of data modeling, schema design, and performance tuning.

Other Skills :

- Familiarity with modern DevOps practices and CI/CD pipelines.

- Experience with monitoring tools and frameworks.

Soft Skills :

- Excellent problem-solving and analytical skills.

- Strong collaboration and communication abilities.

- Detail-oriented with a focus on data quality and consistency.

- Ability to manage multiple tasks in a fast-paced environment.

Preferred Qualifications :

- Experience with other orchestration tools or frameworks.

- Exposure to machine learning workflows and MLOps.

- Certification in AWS or Azure data services is a plus.