Databricks Data Engineer

Job Description

• Databricks Engineers with 4-8 years of professional experience in Databricks, Pyspark, Cloud Platform and SQL.
• Proven experience in designing and implementing data solutions on the Databricks platform with hands on experience in setting up Databricks cluster, working in Databricks modules, data pipelines for ingesting and transforming data from various sources into Databricks
•  Spark-based ETL workflows and data pipelines for data preparation and transformation within Databricks.

Roles & Responsibilities

• Bachelors or Master’s degree with 4-8 years of professional experience.
• Proven experience in designing and implementing data solutions on the Databricks platform.
• Solid understanding of Databricks fundamentals/architecture and have hands on experience in setting up Databricks cluster, working in Databricks modules (Data Engineering, ML and SQL warehouse).
• Configure, set up, and manage Databricks clusters, workspaces, and notebooks to ensure optimal performance, scalability, and resource allocation.
• Design and implement data pipelines for ingesting, cleansing, and transforming data from various sources into Databricks, ensuring data quality, consistency, and reliability. Develop ETL (Extract, Transform, Load) processes as needed.
• Develop, optimize, and maintain Spark-based ETL workflows and data pipelines for data preparation and transformation within Databricks.
• Create and maintain documentation for data pipelines, processes, and best practices.
• Knowledge of Performance Optimization, Monitoring and Automation would be a plus.
• Understanding of data governance, compliance, and security best practices.
• Knowledge in anyone of the cloud platforms AWS, Azure, or GCP is a must.
• Strong Proficiency in Pyspark, Databricks Notebooks, SQL, Python, Scala, or Java.
• Good to have in Databricks Batch process and StreamingShould have good experience in creating Workflows & Scheduling the pipelines.
• Certifications in Databricks would be beneficial.
• Proficiency in SQL for data querying and manipulation.
• Experience with data modeling, ETL processes, and data warehousing concepts.
• Strong problem-solving and troubleshooting skills.
• Excellent communication and collaboration skills.