Senior Data Engineer

About Clairvoyant:

Clairvoyant is a global technology consulting and services company founded in 2012, headquartered in Chandler, US, and has delivery centers across the globe. We help organizations maximize the value of data by providing data engineering, analytics, machine learning, and user experience consulting and development projects to multiple Fortune 500 clients. Clairvoyant clients rely on its deep vertical knowledge and best-in-class services to drive revenue growth, boost operational efficiencies, and manage risk and compliance. Our team of experts with direct industry experience in data engineering, analytics, machine learning, and user experience has your back every step of the way.

Position - Senior Data Engineer

Work Location - Bangalore

Work Timings - 6 AM - 3:30 PM IST

Experience - 8+ Years

Key Responsibilities:

• Design, develop and maintain robust, scalable and efficient data pipelines to move, transform and load data into Snowflake from various data sources, ensuring performance and cost optimization.
• Replicate and optimize SQL Server stored procedures in Snowflake, ensuring functional accuracy and performance in the cloud environment.
• Create and manage data transformations within Snowflake using DBT, building reusable and reliable data models aligned with business requirements.
• Implement and manage automated workflows using Prefect Scheduler to streamline ETL/ELT processes, ensuring reliability, monitoring and alerting for critical workflows.
• Design and maintain CI/CD pipelines using tools such as Jenkins or GitLab, automating testing, deployment and version control.
• Collaborate with cross-functional teams, including data scientists, analysts and business stakeholders, to understand data requirements and deliver impactful solutions, simplifying technical concepts for non-technical stakeholders.
• Use version control systems like GitLab to manage collaborative code development, branching and versioning while fostering best practices in code management.
• Continuously monitor, troubleshoot and optimize data pipelines for performance, scalability and compliance with SLAs.
• Develop reconciliation frameworks and implement automated migration validation processes to ensure data consistency and accuracy during cloud migrations.
• Maintain comprehensive documentation for data pipelines, architectures and processes to enable effective knowledge sharing.

Must-Have:

• 8+ years of hands-on experience as a Data Engineer, with deep expertise in Snowflake, CI/CD practices and DevOps principles.
• Proven experience replicating and optimizing SQL Server stored procedures in Snowflake.
• Strong experience with reconciliation frameworks and automated migration validation for ensuring data consistency during cloud migrations.
• Demonstrated cloud migration expertise, including designing and optimizing scalable data processing solutions.
• Proficiency with CI/CD pipelines using Jenkins, GitLab or similar tools, with integration into DevOps workflows.
• Strong communication and interpersonal skills with the ability to collaborate effectively with technical and non-technical stakeholders.
• Proven ability to manage multiple projects simultaneously with a focus on quality and business value.
• A problem-solving mindset, attention to detail and the ability to thrive in a fast-paced, evolving environment.

Good-to-Have:

• Proficiency in T-SQL for writing complex queries and stored procedures.
• Experience with DBT for building and managing transformations.
• Familiarity with Prefect for orchestrating and automating workflows.
• Hands-on experience with Azure Cloud and cloud-related tools.
• Python experience for data processing or orchestration tasks.

Education:

Bachelor's degree in Computer Science, Software Engineering, MIS or equivalent combination of education and experience