Databricks Consultant

Job Description:

We are seeking an experienced and motivated Databricks Consultant with a strong background in data engineering to join our team. As a consultant, you will work with clients to design, develop, and optimize their data architectures and pipelines using Databricks, empowering them to process and analyze large-scale data efficiently.

Key Responsibilities:

• Databricks Platform Expertise:
  
• Design and implement scalable and efficient data processing solutions on the Databricks platform using Apache Spark, Delta Lake, and MLflow.
  
• Optimize the performance of data pipelines and workflows, ensuring high availability and low-latency data processing.
  
• Data Pipeline Development:
  
• Develop end-to-end ETL/ELT pipelines using Databricks for data ingestion, transformation, and storage in cloud environments (AWS, Azure, or GCP).
  
• Build and maintain data models, ensuring data integrity and consistency across various systems.
  
• Automate data processing workflows to enhance efficiency and reduce manual intervention.
  
• Cloud Infrastructure Management:
  
• Deploy and manage Databricks on cloud platforms (AWS, Azure, GCP) to support data engineering tasks.
  
• Integrate Databricks with cloud storage services (e.g., S3, Azure Data Lake, Google Cloud Storage) and data warehouses (e.g., Snowflake, Redshift, BigQuery).
  
• Collaboration with Stakeholders:
  
• Work closely with data architects, analysts, data scientists, and business teams to understand data needs and translate them into technical solutions.
  
• Provide technical guidance and best practices on data architecture, data pipelines, and Databricks-specific features.
  
• Participate in client workshops and solutioning sessions to design data-driven strategies.
  
• Performance Optimization and Troubleshooting:
  
• Continuously monitor, troubleshoot, and optimize the performance of Databricks clusters and data pipelines.
  
• Identify bottlenecks and implement solutions to enhance performance, scalability, and reliability.
  
• Documentation and Knowledge Sharing:
  
• Create technical documentation, knowledge base articles, and training materials for internal teams and clients.
  
• Provide guidance and mentoring to junior engineers and other team members.
  

Qualifications:

• 5+ years of experience in data engineering, with at least 2+ years of hands-on experience working with Databricks.
  
• Strong experience in Apache Spark, Python, and SQL for data processing and transformation.
  
• Experience with cloud-based technologies and platforms like AWS, Azure, or GCP.
  
• Familiarity with Delta Lake, MLflow, and other Databricks features.
  
• Strong understanding of ETL/ELT processes, data modeling, and data pipeline orchestration.
  
• Expertise in Databricks notebooks, Apache Spark for big data processing, and cloud-based data storage (e.g., AWS S3, Azure Data Lake).
  
• Experience with data engineering tools such as Apache Airflow, dbt, Kafka, and Kubernetes.
  
• Strong SQL skills for querying large datasets and performing data transformations.
  

Preferred Skills:

• Experience in machine learning workflows and integrating ML models with Databricks.
  
• Knowledge of data governance and security best practices in cloud data engineering.
  
• Databricks certification (e.g., Databricks Certified Associate Developer for Apache Spark).
  
• Familiarity with other big data and analytics tools like Hadoop, Presto, Flink, etc.