Azure Databricks Developer

Diverse Lynx is looking forAzure Databricks Developer to join our dynamic team and embark on a rewarding career journey.

• We are seeking a talented and experienced Azure Databricks Engineer/Developer to join our dynamic team
• The ideal candidate will have a strong background in big data analytics, data engineering, and a deep understanding of the Azure Databricks platform
• This role involves designing, developing, and maintaining data processing pipelines and analytics solutions to support our organization's data-driven decision-making

Responsibilities:

• Azure Databricks Development:Design, develop, and implement data processing workflows using Azure Databricks
• Create and optimize Spark jobs for data transformation, cleansing, and aggregation
• Data Engineering:Build and maintain data pipelines to ingest, process, and store large volumes of data
• Collaborate with data scientists and analysts to integrate machine learning models into the data processing workflows
• Performance Tuning:Monitor and optimize the performance of Spark jobs and Databricks clusters
• Troubleshoot and resolve performance bottlenecks in data processing workflows
• Data Integration:Integrate Azure Databricks with other Azure services and data sources
• Ensure seamless data flow and integration between various components of the analytics ecosystem
• Security and Compliance:Implement and enforce security best practices for data storage and processing
• Ensure compliance with data governance policies and regulatory requirements
• Collaboration:Work closely with cross-functional teams, including data scientists, analysts, and business stakeholders, to understand data requirements and deliver solutions
• Documentation:Create and maintain documentation for data pipelines, workflows, and system architecture
• Provide training and knowledge transfer to team members as needed