Data Engineer 2

IMPORTANT - PLEASE APPLY FOR THE ROLE HERE - talent.aidetic.ai//user/application-form

We are seeking a Senior Data Engineer with expertise in Databricks and Big Data technologies. This role will focus on building and optimising scalable data architectures, data pipelines, and processing frameworks using Databricks and Apache Spark, with a strong emphasis on integrating Kafka for real-time data streaming. The ideal candidate should preferably have experience with Azure,. The candidate will work to ensure best practices for data governance, security, and cloud-based big data solutions, with Kafka playing a key role in data streaming and integration.

Compensation - 15 to 20 lacs

Key Responsibilities:

• Design, develop, and implement end-to-end data engineering solutions using Databricks for large-scale data processing and integration projects.
• Build and optimize data ingestion pipelines from various sources, ensuring data quality, reliability, and scalability with Databricks.
• Perform complex data transformation tasks, including data cleansing, aggregation, and enrichment using Databricks and PySpark.
• Integrate and optimize Kafka for real-time data streaming, ensuring seamless data flow through Kafka and Databricks pipelines.
• Collaborate with DevOps and infrastructure teams to optimize the performance and scalability of Databricks clusters and Kafka infrastructure.
• Implement best practices for data governance, data security, and data privacy within the Databricks environment.
• Monitor and troubleshoot data pipelines and Kafka, identifying and resolving performance issues and data quality problems.
• Participate in capacity planning and scaling strategies for both Databricks and Kafka infrastructures to support large-scale data processing.
• Create and maintain documentation for data workflows, processes, and configurations within the Databricks and Kafka environments.
• Understand complex stored procedures for data manipulation and processing, particularly within relational databases such as MySQL and MS SQL.

Requirements:

• 3+ years of experience with Big Data Technologies, including Databricks, Apache Spark, Kafka, Cloud-Native, and Data Lakes.
• Extensive hands-on experience with Databricks for data processing and analytics.
• Strong proficiency in Python, PySpark, Scala, or Java for data processing.
• Proven experience with Kafka for stream processing and real-time data integration.
• Experience working with SQL and relational/NoSQL databases for data extraction and manipulation (Elasticsearch, MySQL, MS SQL).
• Proven experience in comprehending complex stored procedures in SQL for relational databases.
• Preferred experience with Azure, but familiarity with AWS or GCP is acceptable.
• Proven experience in data governance, data security, and data quality best practices within Databricks.
• Experience with Docker, Unix/Linux scripting, and Agile methodologies.
• Strong problem-solving skills and ability to troubleshoot complex data and infrastructure issues.

Additional Skills:

Certifications in Databricks Engineer Professional or Microsoft Azure Data Engineer.