Data Engineer - ETL/SQL/Python (6-10 yrs)

Job Title : Data Engineer

Experience : 6 to 10 years

Location : Bangalore

Key Skills : Databricks, Python, SQL

We are seeking an experienced Data Engineer with strong expertise in Databricks, Python, and SQL to join our team. The primary responsibility of this role is to lead the migration of on-premises big data Spark and Impala/Hive scripts to the Databricks environment. The ideal candidate will have a strong background in data migration projects and experience in transforming and optimizing ETL pipelines for cloud-based architectures.

Key Responsibilities :

- Lead the migration of on-premises big data Spark and Impala/Hive scripts to the Databricks environment.

- Ensure smooth data integration and migration to Azure Cloud, optimizing performance and scalability.

- Build and maintain ETL pipelines, ensuring efficient data transformation and integration with cloud-based architectures.

- Design and develop data models, ETL frameworks, and optimized SQL queries for efficient data processing.

- Work with large-scale data platforms to ensure seamless data accessibility, performance, and security.

- Implement best practices in data governance, data quality, and metadata management.

- Optimize data storage, retrieval, and processing using Azure Data Services.

- Monitor and improve the performance of data pipelines, reducing latency and enhancing data availability.

- Identify and resolve data-related issues, ensuring data consistency, accuracy, and reliability.

- Work closely with DevOps and infrastructure teams to maintain system efficiency.

- Work with cross-functional teams, including data scientists, analysts, and software engineers, to support data-driven decision-making.

- Participate in Agile development cycles, contributing to sprint planning, technical discussions, and knowledge-sharing sessions.

- Document technical processes, best practices, and data transformation workflows for internal teams.

- 6-10 years of hands-on experience in data engineering, data migration, and cloud-based architectures.

- Strong expertise in Databricks, Python, and SQL.

- Experience working with big data technologies such as Apache Spark, Hive, and Impala.

- Hands-on experience with ETL pipeline development and large-scale data processing.

- Proficiency in Azure Cloud Services for data storage, computation, and integration.

- Ability to optimize database queries, data processing pipelines, and data models for efficiency.

- Solid understanding of data warehousing, data governance, and data security best practices.

- Strong analytical and problem-solving skills, with the ability to troubleshoot complex data issues.

- Experience working in Agile development methodologies, including Scrum or Kanban.

- Excellent communication and collaboration skills.