Senior Databricks Engineer

Job Summary:

We are seeking a skilled and experienced Senior Databricks Engineer to join our team. This role is pivotal in designing, developing, and managing scalable data solutions, ensuring seamless data migration, modeling, and analytics within the Databricks environment. The ideal candidate will possess a strong technical background in data engineering, be adept at using Databricks, and thrive in an Agile, collaborative team setting.

Key Responsibilities:

Design, develop, and maintain Python-based data pipelines using Databricks to ensure scalable and efficient data processing
Operationally monitor and maintain data pipelines to ensure reliability and performance.
Collaborate with internal team members to analyze and optimize existing software programs and processes.
Lead data management, integration, analytics, and migration projects, ensuring accuracy and integrity throughout the transition.
Model and manage data within the Databricks environment, aligning with business needs and objectives.
Design and implement advanced analytics and Machine Learning (ML) using Databricks for prediction, forecasting, regression, clustering, classification, etc.
Assist in implementing new technologies to enhance data solutions and processes.
Develop roadmaps and tools to monitor and guide technology usage across projects.
Troubleshoot and resolve data-related challenges, providing innovative technology solutions for complex business problems.
Drive continuous improvement initiatives within the data engineering domain.
Provide routine data extraction and reporting for internal teams and clients.
Document processes, workflows, and solutions to ensure clarity and maintain ability Leverage DevOps and CI/CD best practices to automate the deployment and management of data pipelines and infrastructure

Education and Experience:

Bachelors degree in Computer Science, Programming, or Management Information Systems. At least 3 years of experience in computer science/programming or management information systems, including design, development, implementation, data management and data analytics. Minimum of 2 years of hands-on experience with Databricks
DevSec Ops Experience

Technical Skills:

Proficiency in Python-based data pipeline development
Expertise in data migration, advanced analytics, modeling, and management within Databricks.

Knowledge of DevOps practices and tools.
Experience with Agile methodologies and tools such as Jira.
Familiarity with Amazon S3 and Amazon Quicksight is a plus.

Certifications:

Databricks Certification (preferred).

Soft Skills:

Strong problem-solving and analytical skills.
Excellent communication and collaboration abilities.
Ability to work effectively in a fast-paced, team-oriented environment.