Info Services - Data Engineer - Big Data/Spark (6-9 yrs)

Job Title : Data Engineer with Operations Experience

Location : Hyderabad

Job Description :

We are seeking an experienced Data Engineer to help finalize the setup of our data pipeline and ensure its smooth transition to production. This role is essential in establishing robust operational practices, including monitoring, alerting, and incident management, for maintaining a high-performance pipeline post-launch. The ideal candidate is detail-oriented, proactive, and possesses a strong sense of urgency in handling production environments.

Responsibilities :

- Pipeline Monitoring and Maintenance : Set up and configure monitoring and alerting systems to ensure high data pipeline availability and performance.

- Incident and Change Management : Implement and manage processes for handling pipeline incidents, change requests, and other operational needs.

- Run Book Development : Create comprehensive run books for troubleshooting, incident resolution, and routine operations to support effective pipeline management.

- Operational Readiness : Collaborate with cross-functional teams to ensure the pipeline meets operational requirements and is ready for production.

- Performance Optimization: Identify and address pipeline performance issues, optimizing for scalability and reliability.

- Documentation: Maintain clear, detailed documentation for operational workflows, configurations, and best practices.

Qualifications :

Technical Experience :

- Proven experience with AWS services relevant to data engineering.

- Strong proficiency with Apache Spark for data processing and transformation.

- Familiarity with Grafana or similar dashboarding/visualization tools for monitoring and metrics tracking.

Operational Skills :

- Hands-on experience with monitoring, alerting, and incident management within data pipelines.

- Knowledge of change management and release management processes.

- Experience in writing and maintaining run books and operational documentation.

Personal Attributes :

- A proactive, solution-oriented mindset with a strong sense of urgency.

- Ability to work in a fast-paced, production-oriented environment and manage multiple priorities effectively.

- Excellent problem-solving skills, with a focus on minimizing downtime and ensuring pipeline reliability.

This role is critical to ensuring our data pipeline is production-ready and maintained with best operational practices. If you're a motivated Data Engineer with a passion for operational excellence, we'd love to hear from you!