Director of Data Engineering

What you will do:

• Lead and manage the data engineering team, including hiring, mentoring, performance management, and fostering a culture of innovation and collaboration
• Develop and execute the data engineering strategy, aligning it with overall business objectives and anticipating future data needs
• Architect, design, build, and maintain highly scalable, reliable, and performant data pipelines and infrastructure on AWS, leveraging services like EMR, Glue, Redshift, and others
• Oversee all aspects of data lifecycle management, including data ingestion, transformation, storage (Parquet, Delta), processing (Spark), real-time streaming (Kafka), and efficient retrieval
• Ensure data quality, integrity, and security by implementing robust data validation, monitoring, and governance processes
• Establish and maintain data governance policies and procedures, ensuring compliance with relevant regulations and internal standards
• Evaluate, select, and implement appropriate data engineering tools and technologies, staying up-to-date with industry best practices and emerging trends
• Collaborate closely with data scientists, data analysts, and other stakeholders to understand their data requirements and provide effective data solutions
• Manage budgets, resources, and vendor relationships effectively
• Drive continuous improvement in data engineering processes, promoting automation, CI/CD, and DevOps practices
• Champion data literacy and promote data-driven decision-making across the organization
• Oversee log processing and analysis to gain operational insights and ensure system stability

Who you are:

• Bachelors or Masters degree in Computer Science, Engineering, or a related field
• 10+ years of experience in data engineering, with at least 5 years in a leadership role managing technical teams
• Deep understanding of data warehousing, data lake, and data lakehouse architectures
• Expert-level proficiency in SQL and strong programming skills in Python and ideally Go
• Extensive hands-on experience with big data technologies, including Spark, and real-time streaming platforms like Kafka
• Strong experience with AWS cloud services, particularly EMR, Glue, Redshift, S3, and other relevant services
• Experience with data modeling, ETL/ELT processes, data quality management, and data governance
• Experience working with Parquet and Delta Lake data formats
• Experience with log processing and analysis tools and techniques
• Solid understanding of CI/CD and DevOps practices
• Excellent leadership, communication (written and verbal), and interpersonal skills
• Ability to think strategically, solve complex problems, and work effectively in a fast-paced, dynamic environment
• Experience managing budgets and resources effectively