Data Engineering Lead

Position Overview: We are seeking a skilled and motivated Data Engineer with 6+ years of experience to join our dynamic team. The ideal candidate will have a strong background in Azure Databricks, SQL, and Python, with a proven track record of building and managing robust data pipelines, optimizing data storage, and ensuring the efficient use of cloud resources. This role will be pivotal in designing and implementing scalable data solutions in the Azure ecosystem, also the candidate should have experience in with Selenium Platform is preferred (Not Mandatory)

Key Responsibilities:

• Design, develop, and maintain efficient data pipelines using Azure Databricks to support analytics and reporting.
• Create, optimize, and maintain SQL queries for data extraction, transformation, and loading (ETL).
• Develop custom scripts in Python to automate and enhance data processes.
• Collaborate with cross-functional teams to integrate data solutions that meet business needs.
• Ensure data integrity, quality, and security across multiple sources and environments.
• Troubleshoot and resolve any data or performance issues in the data pipelines.
• Optimize data storage solutions for cost-efficiency and performance in the Azure cloud environment.
• Implement best practices for data governance and security in alignment with industry standards.

Required Skills:

• 4+ years of experience in Data Engineering.
• Proficiency in Azure Databricks for building and managing data pipelines.
• Strong experience with Azure(query writing, optimization, indexing, performance tuning).
• Expertise in Python programming for data-related tasks and automation.
• Hands-on experience with Azure services such as Azure Data Factory, Azure Synapse Analytics, and Azure Storage.
• Strong knowledge of ETL processes and data warehousing concepts.
• Familiarity with version control systems like Git.
• Experience with CI/CD pipelines and DevOps practices in a cloud environment.

Qualifications:

• Experience with big data tools and frameworks like Apache Spark in Databricks.
• Knowledge of data modeling, data lakes, and data governance.
• Exposure to machine learning models and data science workflows.
• Experience in cloud cost management and optimization.
• Bachelors degree in Computer Science, Engineering, or a related field.