Databricks Engineer- AWS

We are looking for an experienced Data Engineer to help us move our clients' analytics platform from AWS Data Platform services to Databricks on AWS using Medallion Architecture. This includes handling large-scale batch and real-time ingestion, implementing Machine Learning (ML) pipelines, and enabling actionable insights through advanced analytics. The ideal candidate will have a strong background in big data technologies, cloud platforms, and machine learning operations.

Required Skills and Experience

1. Technical Expertise

• Proficiency in Databricks Lakehouse features such as Delta Lake, Delta Live Tables (DLT), Auto Loader, Photon, Databricks SQL, and Unity Catalog.
• Expertise in real-time streaming technologies like Kafka, AWS Kinesis, or Databricks Structured Streaming.
• Experience with Google Analytics, WebEngage, and clickstream data analytics on Databricks
• Strong skills in ETL/ELT development
• Experience with Databricks MLflow for machine learning lifecycle management.
• Proficiency in SQL, Python, and Scala for data engineering and ML pipelines.

2. Data Engineering and ML Skills

• Experience designing and deploying end-to-end ML pipelines for large datasets.
• Familiarity with common ML models and how they integrate with operational systems.

3. Cloud and Big Data Knowledge

• Hands-on experience with AWS services such as EC2, S3, RDS and IAM.
• Familiarity with cost optimization strategies in cloud environments.

4. Collaboration Skills

• Ability to work cross-functionally with data scientists, analysts, and business teams to ensure delivery of high-impact analytics and ML solutions.

Preferred Qualifications

• Certifications:
• Databricks Certified Data Engineer Associate/Professional.
• AWS Certified Solutions Architect/Big Data Specialist.