Data Engineer - Generative AI (GenAI) Support

Partner with GenAI Data Scientists to understand their data needs and design efficient data pipelines for data collection, pre-processing, and transformation


Develop and implement data pipelines using tools like Apache Airflow, Luigi, or Prefect to automate data ingestion, cleaning, and feature engineering tasks


Build and manage data infrastructure on the Azure cloud platform to support large-scale GenAI model training and inference


Collaborate with DevOps engineers to ensure seamless integration of data pipelines and infrastructure with the GenAI development lifecycle


Implement data quality monitoring and alerting systems to ensure data integrity for GenAI models


Maintain and optimize existing data pipelines and infrastructure for performance and scalability


Stay up to date on the latest advancements in big data processing technologies and cloud platforms



Qualifications:

Bachelors degree in Computer Science, Information Technology, or a related field (or equivalent experience)


3+ years of experience as a Data Engineer or similar role


Strong experience with data pipelines, data processing tools, and data wrangling techniques


Familiarity with Azure cloud platform is a must


Experience with data warehousing and data lake architectures


Proficiency in SQL and scripting languages and Python


Excellent communication and collaboration skills to work effectively in a cross-functional team


A problem-solving mindset and a passion for building robust data infrastructure



Preferred Skills:

Experience working with Generative AI models or other machine learning projects


Knowledge of data versioning and data lineage management practices