Senior Data Engineer - Azure/PySpark (7-9 yrs)

Job Title : Senior Data Engineer

Experience : 7+ yrs

Location : Remote

Job Description :

We are seeking a highly skilled and experienced Senior Azure Synapse resource with expertise in PySpark, implemented Object Oriented Programming Language in Python and data warehouse design.

This role will be responsible for leading and implementing, optimizing data solutions using Azure Synapse Analytics, with a focus on leveraging PySpark andObject Oriented Programming Language in Python for data processing and analytics.

The ideal candidate will have a strong background in data engineering, hands-on experience with Azure Synapse, Object Oriented Programming Language in Python and the ability to design scalable and efficient data pipelines.

Primary skill : Azure Synapse Analytics as primary cloud service, Pyspark and Object Oriented Programming Language in Python as coding language.

Secondary skill : Spark SQL.

Responsibilities :

- Design and implement data solutions using Azure Synapse Analytics, leveraging best practices and industry standards.

- Develop code by using of class approach via Object Oriented Programming Language in Python.

- Develop scalable and efficient data pipelines in Synapse and use PySpark for data processing, transformation, and analytics.

- Design and implement data models and schemas for efficient storage and retrieval of data.

- Have a detailed understanding of medallion architecture, Knowledge of ER modelling, big data, enterprise data, and physical data models.

- Build ER diagrams, in depth understanding of various schema patterns such as snowflake, star schemas, 3NF, SCD, Data Vaults.

- In depth understanding of various entities such as logical entity, primary key, business key, merge key, foreign key, table grain.

- Collaborate with data engineers, data scientists, and other stakeholders to understand data requirements and design appropriate solutions.

- Optimize and tune data pipelines for performance, scalability, and cost-effectiveness.

- Implement data governance and security controls within Azure Synapse to ensure compliance with regulatory requirements.

- Provide technical leadership and mentorship to junior team members, guiding them in best practices for data engineering and Azure Synapse development.

- Stay current with emerging technologies and trends in data engineering and cloud computing and evaluate their potential impact on our data architecture and solutions.

- Implemented Pytest suites with test driven approach.

Qualifications :

- Bachelor's or Master's degree in Computer Science, Engineering, or a related field.

- 7 to 9 years of experience in data engineering, with a focus on building data solutions in the cloud.

- Strong proficiency in Azure Synapse Analytics including experience with SQL on-demand, Apache Spark pools, and data integration services.

- Expertise in OOPS in Python and Pyspark for data processing, transformation, and analytics.

- Solid understanding of data design principles, including data modelling, schema design, and optimization techniques.

- Experience with Azure data services such as Azure Data Lake Storage, Azure Databricks, and Azure Data Factory, Azure log analytics.

- Strong problem-solving skills and the ability to troubleshoot complex data engineering issues.

- Excellent communication and collaboration skills, with the ability to work effectively with cross-functional teams.

- Azure certifications such as Azure Data Engineer Associate or Azure Synapse Analytics Big Data Engineer are a plus.