Databricks Developer

We are seeking an experienced Databricks Developer / Data Engineer to design, develop, and optimize data pipelines, ETL workflows, and big data solutions using Databricks. The ideal candidate should have expertise in Apache Spark, PySpark, SQL, and cloud-based data platforms (Azure, AWS, GCP). This role involves working with large-scale datasets, data lakes, and data warehouses to drive business intelligence and analytics.

Key Responsibilities:
Design, build, and optimize ETL and ELT pipelines using Databricks and Apache Spark.
Work with big data processing frameworks (PySpark, Scala, SQL) for data transformation and analytics.
Implement Delta Lake architecture for data reliability, ACID transactions, and schema evolution.
Integrate Databricks with cloud services like Azure Data Lake, AWS S3, GCP BigQuery, and Snowflake.
Develop and maintain data models, data lakes, and data warehouse solutions.
Optimize Spark performance tuning, job scheduling, and cluster configurations.
Work with Azure Synapse, AWS Glue, or GCP Dataflow to enable seamless data integration.
Implement CI/CD automation for data pipelines using Azure DevOps, GitHub Actions, or Jenkins.
Perform data quality checks, validation, and governance using Databricks Unity Catalog.
Collaborate with data scientists, analysts, and business teams to support analytics and AI/ML models.
Required Skills & Qualifications:
6+ years of experience in data engineering and big data technologies.
Strong expertise in Databricks, Apache Spark, and PySpark/Scala.
Hands-on experience with SQL, NoSQL, and structured/unstructured data processing.
Experience with cloud platforms (Azure, AWS, GCP) and their data services.
Proficiency in Python, SQL, and Spark optimizations.
Experience with Delta Lake, Lakehouse architecture, and metadata management.
Strong understanding of ETL/ELT processes, data lakes, and warehousing concepts.
Experience with streaming data processing (Kafka, Event Hubs, Kinesis, etc.).
Knowledge of security best practices, role-based access control (RBAC), and compliance.
Experience in Agile methodologies and working in cross-functional teams.
Preferred Qualifications:
Databricks Certifications (Databricks Certified Data Engineer Associate/Professional).
Experience with Machine Learning and AI/ML pipelines on Databricks.
Hands-on experience with Terraform, CloudFormation, or Infrastructure as Code (IaC).