Databricks Data Engineer

*** We are Hiring Immediate Joiners ***

Role : Databricks Data Engineer
Location : Bangalore (Remote)
Experience : 7-12 yrs
Responsibilities:
1. Implement scalable and sustainable data engineering solutions using tools such as Databricks, Azure, Apache Spark, and Python. The data pipelines must be created, maintained, and optimized as workloads move from development to production for specific use cases.
2. Implement Batch and Real-time data ingestion/extraction processes through ETL, Streaming, API, etc., between diverse source and target systems with structured and unstructured datasets
3. Migrate the Stored procedures and functions from Snowflake to Databricks notebook
4. Collaborate with data analysts, reporting team and business advisors to gather requirements and define data models that effectively support business requirements
5. Develop and maintain scalable and efficient data pipelines to ensure seamless data flow across various systems address any issues or bottlenecks in existing pipelines.
6. Monitor data jobs for successful completion. Troubleshoot and resolve any issues that arise to minimize downtime and ensure continuity of data processes.
7. Regularly review and audit data processes and pipelines to ensure compliance with internal standards and regulatory requirements
Familiar with working on Agile methodologies - scrum, sprint planning, backlog refinement etc.
8. Optimize Big Data solutions for high performance and scalability.
Conduct performance optimization, tuning, and troubleshooting of data infrastructure components to ensure optimal performance and resource utilization.
9. Develop and optimize SQL queries for data extraction, transformation, and analysis, ensuring adherence to best practices and performance.
10. Design and implement data models and schemas to support reporting, visualization, and business intelligence initiatives.
11. Monitor and troubleshoot data pipelines and processes, identifying and resolving issues to ensure uninterrupted data flow and availability.
12. Work with ETL tool like Snaplogic to extract data from multiple data sources.

Requirements:
1. 7-12 years' experience on Data Engineering role working with Databricks & Cloud technologies.
2. Ability to work in US PST hours (50% IST and 50% PST time zone)
3. In-depth experience in Databricks core components like Data Frames, Datasets, Spark SQL, Delta Lake, Databricks Notebook, DBFS, and Databricks Connect.
4. Experience in migrating the Snowflake Stored Procs and functions to Databricks notebook
5. Hands on experience using version control systems such as Git and CI/CD workflows and practice
6. Solid understanding of data warehousing concepts, dimensional modeling, and data governance principles.Experience with version control systems (e.g., Git) and CI/CD pipelines for code deployment and automation.
7. Working experience with SnapLogic ETL tool .