Senior Data Engineer

Overview of the role:

Guidepoint seeks an experienced Data Engineer as an integral member of the Pune team. The Pune Technology Hub will serve as the base of our new Data/AI team, dedicated to building a modern data infrastructure for advanced analytics and the development of responsible AI. This strategic investment is integral to Guidepoints vision for the future, aiming to develop cutting-edge Generative AI and analytical capabilities that will underpin Guidepoints Next-Gen research enablement platform and data products.

This role demands proven technical prowess with Lakehouse architecture to drive the development of next-generation research enablement platforms and advanced data products leveraging AI/ML technologies. The Senior Data Engineer will implement Guidepoint’s cloud data infrastructure that will advance BI, Analytics and AI/ML use cases across the organization.

More about Guidepoint’s Technology Team:

Guidepoint’s Technology team thrives on problem-solving and creating happier users. As Guidepoint works to achieve its mission of making individuals, businesses, and the world smarter through personalized knowledge-sharing solutions, the engineering team is taking on challenges to improve our internal application architecture and create new AI-enabled products to optimize the seamless delivery of our services.

Your Day-to-Day Will Include :

• Design and development of systems for the maintenance of the Azure Databricks, ETL processes, business intelligence and data ingestion pipelines for AI/ML use cases.
• Experience in Knowledge Graph Development and experience building Entity relationship and graph-based knowledge models and experience in NLP based Insights.
• Create and support ETL pipelines and tables schemas in order to facilitate the accommodation of new and existing data sources for the Lakehouse on Databricks.
• Maintain data governance and data privacy standards.
• Collaborate with data architects, data scientists, analysts and other business consumers to quickly and thoroughly analyze business requirements to populate the data warehouse, optimized for reporting and analytics.
• Perform root cause analysis on internal and external data and processes to answer specific business questions and identify opportunities for improvement.
• Maintain technical documentation and mentor junior data engineers on best practices in data engineering and Lakehouse architecture.
• Drive innovation and contribute to the development of cutting-edge Generative AI and analytical capabilities for Guidepoint's Next-Gen research enablement platform.

Minimum Qualifications:

• 7+ years of related experience with a bachelor’s degree
• Proven leadership skills, a strategic mindset, and a passion for driving innovation in data engineering and modern data architecture.
• Advanced working SQL knowledge and experience working with relational and NoSQL databases, query authoring (SQL) as well as working familiarity with a variety of databases (e.g. SQL Server, Neo4J, Elastic, Cosmos).
• Experience building and optimizing data pipelines on Azure Databricks.
• In depth knowledge and hands-on experience with Data Engineering, Data Warehousing and Delta Lake on Databricks.
• Highly proficient in Spark, Python, SQL
• Working knowledge of Fivetran is a bonus.
• A successful history of manipulating, processing and extracting value from large, disconnected datasets.
• Exceptional stakeholder management and communication skills to effectively communicate across global teams.

Preferred Qualifications:

• Knowledge of BI Tools like PowerBI etc.
• Experience building and deploying ML and feature engineering pipelines to production using MLFlow.
• Experience with building data pipeline from various business applications like Salesforce, NetSuite, etc.
• Knowledge of message queuing, stream processing, and highly scalable data stores.
• Experience or knowledge of working in a compliance-based environment, including building and deploying compliant software solutions throughout the software life cycle is a nice to have.
• Familiarity with cloud-based AI/ML services and Generative AI tools.