Senior Azure Data Engineer - Spark/Python (8-10 yrs)

Experience : 8 yrs

Job Description :

We are seeking a highly skilled and experienced Senior Azure Data Engineer to join our dynamic team. In this role, you will play a critical part in designing, developing, and implementing cutting-edge data solutions leveraging the power of the Azure platform. You will be responsible for building robust and scalable data pipelines, data lakes, and data warehouses to support critical business decisions within the Financial Services domain.

Key Responsibilities :

- Design, develop, and implement complex data pipelines and ETL/ELT processes using Azure Data Factory (ADF) and Azure Databricks.

- Develop and maintain high-performance data solutions on the Azure platform, including data lakes, data warehouses, and data marts.

- Work with large and complex datasets, including multi-format data, and implement efficient data distribution and integration strategies.

- Leverage Azure Databricks for data processing, transformation, and analysis, utilizing Spark (Scala/Python) for advanced data manipulation.

- Develop and implement data quality checks and validation procedures to ensure data accuracy and integrity.

- Collaborate with data analysts, data scientists, and business stakeholders to understand their requirements and translate them into effective data solutions.

- Contribute to the development and improvement of data governance and data security best practices within the organization.

- Participate in all phases of the software development lifecycle, including design, development, testing, deployment, and maintenance.

- Stay abreast of the latest advancements in cloud computing, data engineering, and big data technologies.

- Mentor junior data engineers and provide guidance on best practices.

Required Skills :

- Strong coding skills in Python, including experience with libraries such as Pandas, NumPy, Scikit-learn. 8+ years of hands-on experience with Azure Data Factory (ADF) and Azure Databricks.

- In-depth knowledge of Azure Storage Gen2, Azure SQL, Azure Functions, and Logic Apps.

- Experience with Azure DevOps for CI/CD pipelines.

- Proficiency in Spark (Scala/Python) for data processing and analysis.

- Experience with large-scale data processing frameworks and tools.

- Proven experience in designing and developing data warehouses and data marts.

- Expertise in ETL/ELT processes and data integration techniques.

- Prior experience with ETL tools such as Informatica, SSIS, or Talend is a plus.

- Proven track record of successful cloud deployments (Azure preferred).

- Experience with agile development methodologies.

- Strong understanding of the Financial Services domain.

- Excellent communication, interpersonal, and collaboration skills.

- Ability to effectively communicate technical concepts to both technical and non-technical audiences.