Sr. Staff Data Engineer

• Use your knowledge of open source, cloud, and data technologies to architect better identity data products for our internal customers.
• Perform architectural reviews for critical business products to ensure they are properly built for reliability, scalability, and cost optimization.
• Design and implement low-latency large-scale data pipelines cleaning and transforming identity data.
• Review and refine the fundamental Data Engineering and SRE practices in the Identity platform and help define a standard for use around Nielsen.
• Mentor other platform team members to increase their technical and soft skills

A LITTLE BIT ABOUT YOU

• You have a genuine passion for data engineering at scale and absolutely love learning new technologies. You don t just want to blindly use platforms and libraries; instead you have a desire to tear them apart and figure out how they work inside. You love scaling and optimizing and seeing just how far or how fast your projects can go. You see all of the building blocks in the community as legos and want to use them to build innovative new solutions at a faster pace than ever.


• You are also familiar with many of the following technologies and programming languages:
• Apache Spark (spark-sql, graphframes)
• Trino/Presto
• Airflow 2.x
• Python
• Scala
• Jupyter (Notebooks)
• Postgres
• AWS S3, EMR, IAM, KMS, EKS, RDS

QUALIFICATIONS

• 5+ years of experience in a MLOps, Data Engineering, DBA or Database Developer role leading critical projects of high complexity.
• Capable of proposing/evaluating data system architectures for high-scale projects.
• In-depth knowledge and experience implementing core big data concepts and tools spanning source control, CI/CD, Infrastructure as Code, monitoring, alerting, logging.
• Deep knowledge of Apache Spark is a must - spark-sql internals specifically is a significant plus.
• Proficiency in AWS, with strong knowledge of S3, KMS and IAM.
• Experience training and mentoring junior team members.
• Experience with Apache Spark graphframes is a plus.