Lead Data Engineer

Greetings from Clairvoyant an EXL Company

We are hiring Lead Data Engineer's at our organization, kindly go through the below job description.

Job Description:

● 8+ years of data management experience.

● Technically strong in Python, PySpark, Hadoop, Hive

● Experienced in data integration projects working for large clients using the above tech stacks. ● Excellent communication and collaboration skills, with the ability to work effectively with cross-functional teams and stakeholders. .

● Replicate and optimize SQL Server stored procedures in Snowflake, ensuring functional accuracy and performance in the cloud environment.

● Implement and manage automated workflows using Prefect Scheduler to streamline ETL/ELT processes, ensuring reliability, monitoring and alerting for critical workflows.

● Design and maintain CI/CD pipelines using tools such as Jenkins or GitLab, automating testing, deployment and version control.

● Collaborate with cross-functional teams, including data scientists, analysts and business stakeholders, to understand data requirements and deliver impactful solutions, simplifying technical concepts for non-technical stakeholders.

● Use version control systems like GitLab to manage collaborative code development, branching and versioning while fostering best practices in code management.

● Continuously monitor, troubleshoot and optimize data pipelines for performance, scalability and compliance with SLAs.

● Develop reconciliation frameworks and implement automated migration validation processes to ensure data consistency and accuracy during cloud migrations.

● Maintain comprehensive documentation for data pipelines, architectures and processes to enable effective knowledge sharing.