Lead/Manager - Data Engineering (8-13 yrs)

Role : Lead / Manager - Data Engineer

Experience : 7 to 9 Years

Work Location : Hyderabad

Job Description :

We are seeking an experienced Lead/Manager - Data Engineer to guide and mentor technical teams in delivering large-scale, data-driven applications. The role requires expertise in building and managing data pipelines, as well as working with modern technologies and frameworks to support scalable and resilient data architectures.

As a Lead/Manager, you will oversee the entire data engineering lifecycle, ensuring the design and implementation of efficient data workflows. You will work closely with cross-functional teams, senior stakeholders, and business leaders to ensure alignment with business objectives and drive successful data initiatives.

Responsibilities :

- Leadership & Team Guidance : Lead and mentor a team of data engineers in building and optimizing data pipelines, frameworks, and services. Ensure efficient collaboration and knowledge sharing across the team.

- Architecture & Scalability : Design and implement scalable, resilient, and high-performance data pipelines and architectures. Leverage microservice principles to build efficient data workflows that support business needs.

- Client Interaction : Engage with senior client stakeholders to understand business requirements, translate them into technical solutions, and ensure that the team's deliverables meet business goals and timelines.

- Data Pipeline Design : Build and maintain ETL/ELT pipelines that process large volumes of structured and unstructured data, optimizing for performance and cost-effectiveness.

- CI/CD & Automation : Set up and maintain continuous integration and continuous deployment (CI/CD) pipelines for automated data processing workflows. Ensure high-quality code through automated testing and deployment practices.

- Monitoring & Observability : Implement robust monitoring and logging solutions (e.g., Prometheus, Grafana) to ensure data pipelines' reliability, performance, and health.

- Security & Compliance : Apply data security best practices in microservices architecture to protect sensitive data. Ensure that the pipelines and systems meet compliance standards.

- Collaboration & Stakeholder Management : Foster strong relationships with business and technical teams, ensuring clear communication and alignment with key business objectives.

Technical Skills :

- Data Engineering Expertise : Strong experience in designing and maintaining large-scale data pipelines, data lakes, and real-time data streaming solutions.

- Python & FastAPI : Expertise in Python, particularly with FastAPI for building high-performance, scalable data services and APIs. Deep understanding of Pydantic for data validation.

- Containerization & Orchestration : Proficient in containerization technologies (Docker, Kubernetes) and experience orchestrating large-scale deployments of data services.

- CI/CD Pipelines : Hands-on experience with CI/CD tools (Jenkins, GitLab CI, etc.) and automation practices to support data pipeline delivery.

- Monitoring & Observability : Skilled in implementing monitoring and logging solutions (e.g., Prometheus, Grafana) to ensure system health, performance, and reliability.

- Data Security : Knowledge of data security best practices, including encryption, secure access controls, and compliance with relevant standards (GDPR, HIPAA, etc.).

- Version Control : Expertise in Git and collaborative workflows to manage code across teams.

Nice to Have :

- Retriever Models & Chunking : Experience with implementing retriever models and optimizing chunking strategies for handling large data volumes efficiently.

- Vector Databases : Familiarity with vector databases and high-performance querying for data-intensive applications such as LLM models and AI-driven solutions.

- Prompt Engineering : Experience with prompt engineering techniques and strategies across diverse data use cases.