Databricks Engineer - Python/Spark (5-6 yrs)

What You'll Do :

- Design and develop data processing pipelines and analytics solutions using Databricks.

- Architect scalable and efficient data models and storage solutions on the Databricks platform.

- Collaborate with architects and other teams to migrate current solution to use Databricks.

- Optimize performance and reliability of Databricks clusters and jobs to meet SLAs and business requirements.

- Use best practices for data governance, security, and compliance on the Databricks platform.

- Mentor junior engineers and provide technical guidance.

- Stay current with emerging technologies and trends in data engineering and analytics to drive continuous improvement.

Requirements :

- Experience with Databricks or similar data processing platforms.

- Strong knowledge of Spark, Scala, Python, and SQL.

- Knowledge on connecting to SAP Hana and extract data into Databricks would be a Plus.

- Proven ability to design and implement scalable data solutions.

- Expertise with code movement process using GIT.

- Experience with ADF Pipelines to execute data bricks workflows.

- Good to have Function knowledge of Supply chain, HR and Payroll data.

- Ability to maintain and extend production data systems to evolve with complex customer needs.

- Excellent problem-solving and analytical skills.

- Strong communication and teamwork abilities.

- Experience supporting project teams of developers and data scientists who build dashboards, reports, and analytics/machine learning models