Big Data Engineer @ Mumbai

Job Title: Product Engineer - Big Data

Location: Mumbai, India

Type: Full-Time

Job Summary:

As a Product Engineer - Big Data, you will be responsible for designing, building, and optimizing large-scale data processing pipelines using the latest Big Data technologies. You will collaborate with cross-functional teams, including data scientists, analysts, and product managers, to ensure data is easily accessible, secure, and reliable. Your focus will be on delivering high-quality, scalable solutions for data storage, ingestion, and analysis, while also driving continuous improvements across the data lifecycle.

Key Responsibilities:

• ETL Pipeline Development & Optimization
  Design and implement complex end-to-end ETL pipelines to handle large-scale data ingestion and processing. Leverage AWS services like EMR, Glue, S3, MSK (Managed Streaming for Kafka), DMS (Database Migration Service), Athena, and EC2 to streamline data workflows and ensure high availability and reliability.
• Big Data Processing
  Develop and optimize real-time and batch data processing systems using Apache Flink, PySpark, and Apache Kafka. Ensure the data is processed in a fault-tolerant manner, with strong focus on scalability and performance. Work with Apache Hudi for managing datasets and enabling incremental data processing and updates.
• Data Modeling & Warehousing
  Design and implement data warehouse solutions that support both analytical and operational use cases. Model complex datasets into optimized data structures, ensuring high performance, easy access, and query efficiency for internal stakeholders.
• Cloud Infrastructure Development
  Build scalable cloud-based data infrastructure leveraging AWS tools. Ensure data pipelines are resilient and adaptable to changes in data volume and variety, with a focus on minimizing costs and maximizing efficiency using services like Managed Apache Airflow for orchestration and EC2 for compute resources.
• Data Analysis & Insights
  Work closely with business teams and data scientists to understand data needs and deliver high-quality datasets. Conduct in-depth analysis to derive insights from the data, identifying key trends, patterns, and anomalies that can drive business decisions. Present findings in a clear and actionable format.
• Real-time & Batch Data Integration
  Enable seamless integration of both real-time streaming & batch data from systems like AWS MSK. Ensure consistency in data ingestion and processing across different formats and sources, providing a unified view of the data ecosystem.
• CI/CD & Automation
  Utilize Jenkins to establish and maintain continuous integration and delivery pipelines. Implement automated testing and deployment workflows, ensuring that new features and updates are seamlessly integrated into production environments without disruptions.
• Data Security & Compliance
  Collaborate with security teams to ensure that data pipelines comply with organizational and regulatory standards, including GDPR, HIPAA, or any other relevant compliance frameworks. Implement data governance frameworks to ensure data integrity, security, and traceability throughout the data lifecycle.
• Collaboration & Cross-Functional Work
  Partner with other engineers, data scientists, product managers, and business stakeholders to understand data requirements and deliver scalable solutions. Collaborate in agile teams, participate in sprint planning, and contribute to architectural discussions.
• Troubleshooting & Performance Tuning
  Identify and resolve performance bottlenecks in data pipelines. Ensure optimal performance through proactive monitoring, tuning, and applying best practices for data ingestion and storage.

Skills & Qualifications:

Must-Have Skills:

1. AWS Expertise:
   Hands-on experience with core AWS services related to Big Data, including but not limited to EMR, Managed Apache Airflow, Glue, S3, DMS, MSK, Athena, and EC2. Deep understanding of cloud-native data architecture.
2. Big Data Technologies:
   Proficiency in PySpark and SQL for data transformations and analysis. Experience working with large-scale data processing frameworks such as Apache Flink and Kafka.
3. Data Frameworks:
   Strong knowledge of Apache Hudi for data lake operations, including CDC (Change Data Capture) and incremental data processing.
4. Database Modeling & Data Warehousing:
   Expertise in designing and implementing scalable data models for OLAP and OLTP systems. Solid understanding of data warehousing best practices and techniques.
5. ETL Pipeline Development:
   Proven experience in building robust, scalable ETL pipelines for processing both real-time and batch data across various platforms.
6. Data Analysis & Insights:
   Ability to conduct complex data analysis to extract valuable business insights. Strong problem-solving skills with a data-driven approach to decision-making.
7. CI/CD & Automation:
   Basic to intermediate knowledge of CI/CD pipelines using Jenkins or similar tools to automate deployment and monitoring of data pipelines.

Preferred Skills:

• Experience with containerization and orchestration tools like Docker and Kubernetes for data infrastructure deployment.
• Familiarity with data governance frameworks and tools to ensure compliance and security.
• Knowledge of monitoring tools such as AWS CloudWatch, Splunk or Prometheus to keep track of the health and performance of data systems.