Data Engineer - PySpark/Databricks (5-10 yrs)

Job Description :

- Design, develop, and implement data pipelines using PySpark and Databricks.

- Extract, transform, and load data from various sources into data warehouses and data lakes.

- Optimize data ingestion and processing performance.

- Develop and maintain data quality checks and monitoring systems.

- Collaborate with data analysts and data scientists to understand their data needs.

- Implement data security and privacy best practices.

- Automate data pipelines using tools like Airflow.

Required Skills and Experience :

- 5+ years of experience in data engineering.

- Strong proficiency in PySpark and Databricks.

- Experience with ETL processes and data ingestion techniques.

- Knowledge of data warehousing and data lake concepts.

- Experience with data quality and validation techniques.

- Familiarity with cloud platforms (i.e., AWS, Azure, GCP) is a plus.

- Strong problem-solving and analytical skills.

- Excellent communication and collaboration skills.

Nice-to-Have Skills :

- Experience with Jenkins, GitHub Actions, and CI/CD pipelines.

- Knowledge of Agile frameworks (Scrum, Kanban).

- Experience with performance engineering and optimization techniques.

- Familiarity with Terraform for infrastructure as code