Senior Data Engineer - Python/PySpark (7-10 yrs)

Senior Data Engineer

Key Responsibilities :

- Design, develop, and implement data pipelines and data processing solutions using Python/PySpark.

- Perform data modeling, data analysis, and data transformation tasks.

- Utilize Azure Databricks, Azure Data Factory, Azure Data Lake, and Azure DevOps to build and deploy data solutions.

- Ensure data quality and security by implementing appropriate measures and adhering to best practices.

- Work with relational databases such as SQL Server, Oracle, and MySQL.

- Develop and maintain CI/CD pipelines for data engineering projects.

- Collaborate effectively with data scientists, data analysts, and other stakeholders.

- Stay abreast of the latest advancements in data engineering technologies and best practices.

Qualifications :

- 7-10 years of experience in data engineering or a related field.

- Strong proficiency in Python and PySpark.

- Expertise in Azure Databricks, Azure Data Factory, Azure Data Lake, and Azure DevOps.

- Experience with relational databases (SQL Server, Oracle, MySQL).

- Strong understanding of data warehousing and data lake concepts.

- Experience with CI/CD principles and tools.

- Knowledge of API/web service security and data security practices.

- Excellent problem-solving, analytical, and communication skills.

- Ability to work independently and as part of a team.

Preferred Qualifications :

- Experience with data visualization tools ( Power BI, Tableau)

- Experience with Streamlit or RShiny for building data applications