Data Engineer - Databricks (4-8 yrs)

Job Description :

We are seeking a highly skilled and motivated Data Engineer with expertise in Databricks to join our dynamic team. In this role, you will be responsible for designing, developing, and optimizing data pipelines, enabling the seamless integration and processing of large datasets. The ideal candidate will have a solid understanding of cloud data platforms, data engineering best practices, and the ability to work with cutting-edge technologies in a collaborative and fast-paced environment.

Key Responsibility :

- Design, build, and maintain scalable, high-performance data pipelines in Databricks to support analytics and machine learning workflows.

- Collaborate with data scientists, analysts, and other engineers to define and implement data solutions.

- Develop ETL processes for ingesting, transforming, and storing data across a variety of data sources, ensuring data quality and integrity.

- Optimize and tune existing Databricks pipelines and workflows for performance, scalability, and reliability.

- Build and maintain data models and data warehouses, leveraging tools such as Delta Lake and Apache Spark.

- Write efficient SQL queries for data analysis and reporting.

- Automate data workflows, monitor pipeline performance, and implement proactive solutions to resolve any issues.

- Ensure data security, privacy, and compliance by adhering to best practices and organizational policies.

- Participate in code reviews, mentorship, and knowledge-sharing sessions with the team.

- Troubleshoot and debug complex data issues to ensure the robustness and efficiency of data solutions.