Data Engineer - ETL/PySpark (6-12 yrs)

Job Description :

We are looking for an experienced Data Engineer to join our team. The ideal candidate will have strong expertise in data engineering technologies and tools, with a focus on cloud platforms like AWS and Azure, along with data processing frameworks such as Databricks, Airflow, PySpark, and Python.

Key Responsibilities :

Data Pipeline Development :

- Design, develop, and maintain scalable data pipelines for ingestion, transformation, and storage.

- Utilize Databricks, Airflow, and PySpark to build high-performance data workflows.

Cloud Platform Expertise :

- Leverage AWS and Azure cloud services for data processing, storage, and analytics.

- Optimize and manage data lakes and warehouses for performance and cost-efficiency.

Data Integration and ETL :

- Implement ETL processes to integrate data from various sources into centralized systems.

- Ensure data quality, integrity, and consistency across pipelines.

Performance Optimization :

- Optimize data workflows and queries for performance improvements and reduced latency.

- Implement robust error handling and monitoring for data workflows.

Collaboration :

- Work closely with cross-functional teams, including Data Scientists, Analysts, and Business Stakeholders.

- Translate business requirements into technical solutions and data workflows.

Documentation and Reporting :

- Maintain comprehensive documentation of data workflows, pipelines, and systems.

- Provide insights and reporting on data performance and pipeline status.

Key Skills & Requirements :

Technical Expertise :

- Strong proficiency in AWS and Azure cloud platforms.

- Experience with Databricks, Airflow, PySpark, and Python.

- Hands-on experience with data modeling, transformation, and analytics.

Data Processing :

- Expertise in building scalable and reliable ETL/ELT workflows.

- Knowledge of big data tools and techniques for processing large datasets.

Performance Optimization :

- Experience in optimizing data workflows, queries, and cloud resource usage.

Soft Skills :

- Strong problem-solving and analytical abilities.

- Excellent communication skills to liaise with technical and non-technical stakeholders.

Preferred Qualifications :

- Experience with Snowflake, Redshift, or similar data warehouses.

- Familiarity with CI/CD pipelines for data workflows.

- Exposure to machine learning workflows and MLOps.