Senior Data Engineer - Azure Databricks (2-7 yrs)

Job Description :

We are seeking a Senior Data Engineer with expertise in Databricks and Big Data technologies. This role will focus on building and optimising scalable data architectures, data pipelines, and processing frameworks using Databricks and Apache Spark, with a strong emphasis on integrating Kafka for real-time data streaming. The ideal candidate should preferably have experience with Azure,. The candidate will work to ensure best practices for data governance, security, and cloud-based big data solutions, with Kafka playing a key role in data streaming and integration.

Key Responsibilities :

- Design, develop, and implement end-to-end data engineering solutions using Databricks for large-scale data processing and integration projects.

- Build and optimize data ingestion pipelines from various sources, ensuring data quality, reliability, and scalability with Databricks.

- Perform complex data transformation tasks, including data cleansing, aggregation, and enrichment using Databricks and PySpark.

- Integrate and optimize Kafka for real-time data streaming, ensuring seamless data flow through Kafka and Databricks pipelines.

- Collaborate with DevOps and infrastructure teams to optimize the performance and scalability of Databricks clusters and Kafka infrastructure.

- Implement best practices for data governance, data security, and data privacy within the Databricks environment.

- Monitor and troubleshoot data pipelines and Kafka, identifying and resolving performance issues and data quality problems.

- Understand complex stored procedures for data manipulation and processing, particularly within relational databases such as MySQL and MS SQL.

Requirements :

- 3+ years of experience with Big Data Technologies, including Databricks, Apache Spark, Kafka, Cloud-Native, and Data Lakes.

- Extensive hands-on experience with Databricks for data processing and analytics.

- Strong proficiency in Python, PySpark, Scala, or Java for data processing.

- Proven experience with Kafka for stream processing and real-time data integration.

- Experience working with SQL and relational/NoSQL databases for data extraction and manipulation (Elasticsearch, MySQL, MS SQL).

- Proven experience in comprehending complex stored procedures in SQL for relational databases.

- Preferred experience with Azure, but familiarity with AWS or GCP is acceptable.

- Experience with Docker, Unix/Linux scripting, and Agile methodologies.

Additional Skills :

- Certifications in Databricks Engineer Professional or Microsoft Azure Data Engineer.