Data Engineering Lead - X Delivery

• Leading and Delivering Analysis: Leading data engineering initiatives by providing original analysis and insights to project teams. You will own and oversee analytics modules, ensuring seamless integration with case teams.
• Establishing Thought Leadership: Building credibility as a thought leader by partnering with case teams on complex analytics topics. You will draw insights and conclusions on a broad range of external and internal data engineering challenges.
• Communicating Insights: Effectively communicating analytical insights through sophisticated synthesis, packaging results in presentations, reports, and visualizations (PPT slides, charts). You will guide the team in collecting, synthesizing, and analyzing learnings to develop best practices and methodologies.
• Managing Documentation and Compliance: Ensuring all deliverables receive proper sign-off before being uploaded to internal repositories. You will sanitize and secure confidential client information before inclusion in marketing and reference materials.
• Developing Expertise: Developing deep expertise in key analytics topics or platforms, guiding the team in the execution of analytical approaches, and contributing to the selection and implementation of innovative solutions.

• Big Data Tools & Frameworks: Selecting, integrating, and overseeing the implementation of Big Data tools and frameworks required to meet project requirements. You will ensure the seamless operation of these tools across on-premise and cloud architectures.
• ETL Processes: Leading the design and implementation of ETL processes across various architectures, optimizing for performance and efficiency.
• Performance Monitoring & Optimization: Proactively monitoring system performance, advising on and implementing necessary infrastructure changes to enhance performance and scalability.
• Vendor Management: Managing relationships with vendor teams, coordinating efforts to ensure alignment with project objectives. You will oversee vendor communications with different stakeholders, ensuring that expectations are clearly communicated and met.

• Leadership & Communication: Demonstrating clear and confident leadership, able to communicate complex ideas effectively. You will deliver messages in a concise manner with strong written and verbal communication skills, guiding and mentoring junior engineers.
• Analytical Thinking: Exhibiting strong analytical thinking and solutioning abilities with hands-on experience in delivering advanced analytics projects. You will provide expert analytical advice and contribute to the knowledge base of the team.

• Educational Background: Bachelors or masters degree in Computer Science, Engineering, or a related field.
• Experience: 6-10 years of experience in Data Engineering across various industries, with proven expertise in delivering analytics solutions in commercial settings. Prior consulting experience is a plus.

Technical Expertise:

• Proficient understanding of distributed computing principles.
• Expertise in managing and optimizing Spark clusters, along with other implementations of Spark.
• Strong problem-solving skills, particularly in optimizing cluster operations.
• Deep knowledge of cloud ecosystems (AWS, Azure, Google Cloud, IBM Cloud), with expertise in at least one.
• Experience in building and leading stream-processing systems using technologies like Storm or Spark-Streaming.
• Proficiency in Big Data querying tools (Pig, Hive, Impala).

Preferred Skills:

• Experience in integrating data from multiple sources.
• Familiarity with NoSQL databases (eg, HBase, Cassandra, MongoDB).
• Knowledge of ETL techniques and frameworks (eg, Nifi, Flume).
• Experience with messaging systems (eg, Kafka, RabbitMQ).
• Familiarity with Big Data ML toolkits
• Understanding of Lambda Architecture, including its advantages and trade-offs.

• Team Management: Leading a team of data engineers, providing mentorship, guidance, and performance feedback. You will foster a collaborative and innovative team culture.
• Project Ownership: Taking full ownership of data engineering projects, from conception through execution to delivery, ensuring alignment with business objectives and client expectations.
• Vendor & Stakeholder Management: Managing vendor teams and maintaining effective communication with various stakeholders. You will ensure that vendor contributions align with project goals and stakeholder expectations.
• Strategic Vision: Contributing to the strategic vision of the data engineering function, identifying opportunities for innovation and continuous improvement