Data Engineer - Predictive Modeling (8-13 yrs)

Job Description :

We are seeking a highly skilled Data Engineer with 6+ years of experience in data engineering best practices and a proven track record of building or significantly contributing to the development of data platforms and AIOps platforms from scratch.

The ideal candidate must be proficient in developing end-to-end data engineering solutions and building the core platform features, including data preparation, schema-on-read, schema-on-write, data lake, data warehouse, ETL pipelines, headless architecture, microservices, and AIOps capabilities.

The candidate must possess strong expertise in :

- Data Retention Management: Designing and managing data retention policies within data lakes and data warehouses.

- Pipeline Mechanisms: Creating consumable mechanisms for end-users to build custom data pipelines.

- Data Ingestion: Facilitating seamless ingestion of raw data into data lakes and processed data into data warehouses.

- Data Mart Development: Establishing procedures and mechanisms to empower end-users to build data marts on top of the data warehouse, serving as a foundation for Data-as-a-Service (DaaS) and enabling AIOps functionalities.

Key Responsibilities :

- Platform Development: Build and contribute to scalable data and AIOps platforms supporting ingestion, preparation, transformation, observability, and automation.

- Data Lake Architecture: Design protocols to ingest batch/live data into elastic data lakes and integrate with external data lake and data warehouse providers.

- AIOps Features: Build or enhance features such as anomaly detection, predictive analytics, root cause analysis, event correlation, and intelligent alerting.

- Data Connections: Implement operators and mechanisms to enable file uploads, API connectors, message queues, cloud storage, and IoT stream ingestion.

- Data Processing: Build robust procedures for data cleaning, transformation, enrichment, validation, aggregation, classification, and anonymization.

- Data Destinations: Develop features for exporting data to warehouses, APIs, message queues, analytics tools, and visualization dashboards.

- ETL Pipelines: Create scalable ETL pipelines for seamless data integration and transformation.

- Open-Source Integration: Utilize open-source tools and frameworks for real-time processing, automation, and observability.

- Microservices: Ensure modular, scalable design using headless architecture and microservice-driven approaches.

- Collaboration: Work closely with DevOps, SRE, and cross-functional teams to align data engineering with platform observability and automation.

- Governance: Implement robust data governance protocols to ensure security, quality, and compliance.

Mandatory Skills and Qualifications :

Education :

- Minimum Bachelor's degree in Computer Science, Electronics and Communication Engineering (ECE), or Information Technology (IT) from a recognized institution.

Technical Skills :

1. Programming: Proficiency in Python, Java, or Scala.

2. Databases: Expertise in relational (MySQL, PostgreSQL, SQL Server) and NoSQL databases (MongoDB, Cassandra).

3. Data Warehousing & ETL Tools: Experience with tools like Amazon Redshift, Talend, Informatica, or Apache Airflow.

4. Data Lake Management: Strong expertise in data retention policies and lifecycle management in data lakes.

5. Cloud Platforms: Hands-on experience with AWS, Azure, and GCP.

6. Open-Source Frameworks: Proficiency with Apache Spark, Kafka, Flink, Druid, and Presto for data processing and orchestration.

7. AIOps Tooling: Familiarity with tools like Prometheus, Grafana, Elasticsearch, and Fluentd for observability and monitoring.

8. Data-as-a-Service (DaaS): Proven experience in designing and exposing data marts as services.

9. Microservices and Architecture: Hands-on experience in implementing headless architecture for scalable and extensible platforms.

10. Data Visualization: Proficiency with tools like Tableau and Excel.

11. Machine Learning: Foundational knowledge of ML principles for integration with AIOps features.

Core Platform Features Knowledge:

- Data Connections: File upload, API connector, message queue connector, cloud storage, IoT stream ingestion.

- Data Processing: Real-time processing, data normalization, machine learning integration, and data classification.

- Data Destinations: Cloud storage, cold storage archiving, data warehouse writing, and dashboard building.

AIOps Features :

- Intelligent alerting mechanisms.

- Event correlation and anomaly detection.

- Predictive analytics for proactive issue resolution.

- Root cause analysis for faster troubleshooting.

Soft Skills :

- Strong critical thinking and problem-solving skills.

- Excellent communication and collaboration abilities.

- Effective time management to handle multiple priorities and deadlines.