Lead Data Engineer (5-9 yrs)

Senior Lead Data Engineer, will play a pivotal role in designing, developing, and maintaining data infrastructure. You will collaborate with cross-functional teams, and ensure the seamless integration of data solutions across multiple platforms. Your expertise in Python or Java, combined with hands-on experience in various data platforms, cloud DevOps, and analytics, will be crucial in driving our data initiatives forward. Experience with AI, particularly Large Language Models (LLMs), is highly desirable and considered a plus.

Key Responsibilities :

- Mentor a team of data engineers, fostering a culture of continuous improvement and innovation.

- Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver effective solutions.

- Participate in strategic planning and contribute to the overall vision of the data platform architecture.

- Data Platform Development

- Design, develop, and maintain scalable data pipelines and ETL processes using Python or Java.

- Implement and manage data storage solutions across multiple platforms (e.g., SQL/NoSQL databases, data lakes, data warehouses).

- Ensure data quality, integrity, and security across all data platforms.

- Hands-on implementation and optimization of data processing frameworks (e.g., Apache Spark, Kafka).

- Integrate cloud services (AWS, Azure, GCP) into the data infrastructure, leveraging cloud-native tools and services.

- Implement DevOps practices for continuous integration and delivery (CI/CD) in data engineering workflows.

- Analytics & AI Integration

- Support analytics teams by providing reliable and efficient data access and processing capabilities.

- Explore and integrate AI/ML solutions, particularly leveraging Large Language Models (LLMs), to enhance data-driven applications.

- Performance Optimization & Troubleshooting

- Monitor and optimize the performance of data pipelines and storage solutions.

- Troubleshoot and resolve data-related issues promptly to ensure minimal downtime and data reliability.

Qualifications :

- 5-9 years of hands-on experience in data engineering, with a focus on Python or Java development.

- Proficient in Python or Java programming languages.

- Extensive experience with data integration, ETL processes, and data pipeline orchestration tools (e.g., Apache Airflow, Luigi).

- Strong knowledge of various data platforms, including relational and NoSQL databases, data lakes, and data warehouses.

- Experience with cloud platforms (AWS, Azure, GCP) and cloud-based data services.

- Familiarity with DevOps practices and tools (e.g., Docker, Kubernetes, Jenkins, Terraform).

- Understanding of data modelling, data architecture, and data governance principles.

- Analytics & AI

- Solid understanding of data analytics concepts and ability to support analytics workflows.

- Exposure to AI/ML technologies, with experience in integrating AI solutions being a significant advantage.

- Strong leadership and team management abilities.

- Excellent problem-solving and analytical skills.

- Effective communication and collaboration skills, with the ability to work cross-functionally.

- Ability to manage multiple projects simultaneously and meet deadlines.

Preferred Qualifications :

- Experience with Big Data technologies such as Hadoop, Hive, or similar.

- Knowledge of real-time data processing and streaming platforms (e.g., Apache Kafka, Flink).

- Familiarity with infrastructure as code (IaC) tools like Terraform or CloudFormation.

- Experience with version control systems (e.g., Git) and agile development methodologies.

- Understanding of security best practices in data engineering and cloud environments.

- Prior experience working with AI frameworks and Large Language Models (LLMs).