Job Summary: We are seeking a talented and motivated Data Engineer with experience in Databricks, Azure, and big data technologies. The ideal candidate should have strong expertise in building scalable ETL pipelines using Python and Scala and be familiar with Azures data services ecosystem. You will play a key role in managing, transforming, and analyzing large datasets to drive business insights.
Key Responsibilities:
Design, build, and maintain ETL pipelines using Databricks in Azure.
Work with large-scale data in distributed systems to ensure data quality, integrity, and reliability.
Develop and implement scalable data models and architectures. Optimize existing data workflows for performance and scalability. Collaborate with data scientists, analysts, and stakeholders to understand data requirements. Monitor, troubleshoot, and improve the performance of data pipelines. Integrate data from various sources and ensure proper data governance and security.
Utilize Azure services such as Data Lake, Azure SQL Database, Azure Data Factory, and more for end-to-end data solutions.
Qualifications: Bachelor s degree in Computer Science, Information Technology, or a related field.
3-5 years of experience in data engineering roles, preferably with Databricks and Azure.
Proficiency in Python and Scala for building ETL processes. Strong knowledge of big data technologies like Hadoop, Spark, and distributed systems.
Hands-on experience with Azure data services (e.g., Data Lake, Data Factory, Databricks).
Experience with SQL and NoSQL databases. Familiarity with data governance and best practices for managing secure data pipelines. Strong problem-solving skills and attention to detail. Ability to work in an agile, fast-paced environment.
Preferred Qualifications:
Experience with other cloud platforms (AWS, GCP) is a plus.