Databricks Lead

Job Description
Role: Databricks Lead Resource / Level 6
Location: Offshore/India
Who are we looking for?
Looking for a Databricks Lead resource (with experience in AWS) with 5+ years of relevant experience and with overall experience of 8-10 years of experience.
Role & Responsibilities Overview:

• Develop and optimize ETL pipelines from various data sources using Databricks on cloud (AWS, Azure, etc.)
• Experienced in implementing standardized pipelines with automated testing, DevOps for CI/CD, Terraform for infrastructure as code, and Ganglia for monitoring.
• Continuously improve systems through performance enhancements and cost reductions in compute and storage
• Develop and maintain data pipelines for effective data extraction transformation and loading.
• Data Processing and API Integration: Utilize Spark Structured Streaming for real-time data processing and integrate data outputs with REST APIs
• Develop and maintain Scalable Architecture database design and data pipelines.
• Data Engineering: Develop and optimize data pipelines for ingesting, processing, and transforming large volumes of data from diverse sources using Databricks and related technologies.
• Performance Optimization: Optimize Databricks jobs, queries, and workflows for performance, scalability, and cost-efficiency, leveraging best practices and optimization techniques.
• Lead Data Engineering Projects to manage and implement data-driven systems.
• Develop and maintain data quality standards.
• Implement security controls, access management policies, and data governance frameworks within Databricks to ensure data privacy, compliance, and regulatory requirements.
• Integrate data across different systems and platforms.
• Production deployment experience with data governance solutions and hands-on experience with cloud data lakes.

Technical Skills:

• Experience in developing and implementing ETL pipelines from various data sources using Databricks on cloud (AWS, Azure)
• Proven experience designing, implementing, and managing data analytics using Databricks.
• Strong proficiency in Apache Spark, SQL, Python with hands-on experience developing data pipelines.
• Technologies - IaaS (AWS or Azure or GCP), Databricks platform, Delta Lake storage, Spark (PySpark, Spark SQL).
• Ability to work independently with some level of ambiguity and juggle multiple demands.
• Understand and document data transformation requirements.
• Provide technical guidance and mentorship to junior team members.
• Experience working in Insurance domain .
• Data Modeling/ Data Lineage and awareness of Canonical data model implementation

Good to have:

• Experience in Airflow, Splunk, Power BI, Git, Azure Devops

Behavioral Skills:

• Must have excellent communication skills to closely work with Customers.
• Resolve technical issues of projects.
• Participates as a team member.
• Effectively collaborates and communicates with the stakeholders and ensure client satisfaction.

Qualification:

• Somebody who has at least 8+ years of experience in Databricks Implementation / Development.
• Good to have Power BI Architecture Experience / Exposure.
• Experience of working in Cloud Platforms like Azure, AWS and using services like Data Factory, Databricks, Glue, S3, Athena
• Education qualification: Any bachelor’s degree from a reputed college.

Talent Fulfillment Group JD Format