Databricks Engineer

Overview:
We are looking for an experienced Databricks Engineer with 4-5 years of hands-on experience in designing, developing, and managing data solutions using Databricks, Apache Spark, and cloud technologies. As a key member of our data engineering team, you will work on building scalable data pipelines, optimizing data workflows, and supporting the integration of Databricks with cloud platforms to support our data-driven initiatives.

Key Responsibilities

• Design, develop, and manage end-to-end data pipelines using Databricks and Apache Spark to process large-scale datasets efficiently.
• Collaborate with cross-functional teams, including data scientists, analysts, and cloud architects, to design and implement data models and analytics solutions.
• Create and maintain Databricks Notebooks for data exploration, analysis, transformation, and reporting.
• Optimize the performance of data pipelines and Spark jobs, ensuring efficient resource utilization and minimal processing time.
• Deploy and manage Databricks clusters, ensuring high availability, scalability, and security for data workloads.
• Build and integrate ETL processes for seamless data ingestion, cleansing, and transformation within Databricks.
• Monitor and troubleshoot production workflows and Databricks jobs, resolving performance issues and addressing data quality concerns.
• Work with cloud platforms (AWS, Azure, or GCP) to integrate Databricks with cloud-native storage and compute services.
• Collaborate with data governance and security teams to ensure data compliance and privacy requirements are met.
• Document best practices, processes, and workflows for Databricks and data engineering projects.

Qualifications

• Bachelor s or Master s degree in Computer Science, Engineering, Data Science, or a related field.
• 4-5 years of hands-on experience working with Databricks, Apache Spark, or similar big data processing tools.
• Strong programming skills in Python or Scala, with experience in building data pipelines and optimizing performance.
• Proficient in writing SQL for querying and transforming large datasets.
• Solid experience with cloud platforms (AWS, Azure, or GCP) and integrating Databricks with cloud-native services (e.g., S3, Azure Blob Storage, Redshift, BigQuery).
• Experience in Databricks workspace, Notebooks, Cluster Management, and Delta Lake for building reliable data workflows.
• Solid understanding of data engineering concepts, such as ETL processes, data modeling, and distributed computing.
• Experience in data pipeline orchestration and scheduling tools like Apache Airflow or Databricks jobs.
• Ability to troubleshoot and resolve performance bottlenecks, including Spark jobs and Databricks workloads.
• Excellent communication skills and ability to collaborate across teams to define requirements and deliver solutions.