AWS Databricks Engineer - SQL (5-8 yrs)

Job Title : AWS Databricks Engineer (SQL).

Location : Bangalore.

Experience : 5-8 Years.

Job Description :

As an AWS Databricks Engineer (SQL), you will play a key role in the architecture and development of cloud-based data pipelines and solutions on the AWS platform using Databricks. You will work closely with data engineers, analysts, and business teams to create efficient, scalable, and robust data solutions. Your expertise in SQL and Databricks will enable you to manage and process large datasets, ensuring that data is ready for analysis and consumption across various business units.

Key Responsibilities :

- Databricks Development : Design, develop, and maintain data pipelines using AWS Databricks, integrating data from multiple sources for efficient processing and analysis.

- SQL Expertise : Write complex SQL queries to extract, transform, and load (ETL) data from various data sources, ensuring data accuracy and consistency.

- Data Processing : Utilize Databricks to run Spark-based data processing jobs, optimize workflows, and process large datasets in a distributed environment.

- Data Architecture : Collaborate with data architects to design and implement data storage and processing architectures on AWS, ensuring scalability and performance.

- Optimization : Monitor and optimize SQL queries and Databricks workflows for performance and efficiency in the cloud environment.

- Cloud Infrastructure : Leverage AWS services such as S3, Redshift, and Lambda in conjunction with Databricks to build scalable data pipelines.

- Data Governance and Quality : Implement data governance and data quality best practices, ensuring compliance and data accuracy across all data pipelines.

- Collaboration and Documentation : Work closely with cross-functional teams to understand data requirements and document processes, workflows, and technical solutions.

- Troubleshooting and Issue Resolution : Identify and resolve issues related to performance, data inconsistencies, and system errors across Databricks and SQL-based solutions.

Required Skills And Qualifications :

- AWS Databricks : Hands-on experience with Databricks on AWS, including knowledge of Spark and Delta Lake.

- SQL : Strong proficiency in SQL for querying, transforming, and managing data in relational databases.

- AWS Services : Experience working with AWS data services such as S3, Redshift, and Glue, in conjunction with Databricks.

- Data Pipeline Development : Experience building and optimizing data pipelines, ETL processes, and batch processing workflows.

- Big Data Processing : Knowledge of distributed computing frameworks (like Apache Spark) for large-scale data processing.

- Cloud Architecture : Strong understanding of cloud-based data architecture and best practices for building scalable data systems on AWS.

- Performance Optimization : Experience optimizing SQL queries and Databricks jobs for improved speed and resource utilization.