Senior Data Engineer - Databricks

Design, develop, and implement Databricks solutions to support data integration, analytics, and business intelligence requirements.
Collaborate with data architects, data scientists, and data analysts to understand data requirements and to ensure Databricks solutions meet business needs.
Manage and optimize Databricks environments, including performance tuning, scaling, and cost management.
Develop and maintain CI/CD pipelines for Databricks projects using tools like Jenkins, GitHub Actions, or Azure DevOps.
Implement infrastructure as code (IaC) solutions using Terraform to automate the deployment and management of Databricks resources on Azure Cloud.
Perform data cleansing, transformation, and integration tasks to ensure data quality and integrity within Databricks.
Integrate third-party application data into Databricks, ensuring seamless data flow and compatibility.
Monitor and troubleshoot Databricks jobs, clusters, and workflows, and provide technical support and guidance to other team members.
Evaluate and recommend new tools, technologies, and best practices to enhance Databricks implementations and data management processes.
Develop and maintain documentation for Databricks solutions, including architecture diagrams, data flow diagrams, and technical specifications.
Stay current with industry trends, emerging technologies, and best practices in Databricks, data engineering, and data management.