Senior Data Engineer - Azure Databricks (7-8 yrs)

Responsibilities :

- Design, develop, and implement scalable data pipelines using Databricks and PySpark for data ingestion, transformation, and loading (ETL) processes.

- Develop and maintain data models that ensure data quality, consistency, and efficiency.

- Utilize SQL for data extraction, transformation, and analysis.

- Implement data integration strategies to seamlessly connect various data sources.

- Design and implement data architecture solutions to optimize data storage, processing, and access.

- Ensure data governance, security, and compliance by adhering to established policies and procedures.

- Monitor and optimize data pipelines for performance and scalability.

- Collaborate with data analysts, data scientists, and other stakeholders to understand data requirements and translate them into technical solutions.

- Document data pipelines and data models for clarity and maintainability.

- Stay up-to-date with the latest advancements in data engineering technologies and best practices.

Qualifications :

- 7+ years of experience in data engineering or a related field.

- Proficiency in SQL, data modeling techniques, and ETL tools.

- Strong knowledge of data integration, data migration strategies, and data architecture design principles.

- In-depth expertise in Databricks and PySpark for large-scale data processing.

- Solid understanding of data governance, data security best practices, and compliance regulations.

- Excellent problem-solving, analytical, and communication skills.

- Ability to work independently and as part of a cross-functional team.

- Strong attention to detail and a commitment to delivering high-quality work