2 mFilterIt Jobs

Data Engineer (3-4 Years Experience) - Real-time & Batch Processing | AWS, Kafka, Click House, Python

Location : NOIDA.

Experience : 3-4 years.

Job Type : Full-Time.

About the Role :

We are looking for a skilled Data Engineer with 3-4 years of experience to design, build, and maintain real-time and batch data pipelines for handling large-scale datasets.

You will work with AWS, Kafka, Cloudflare Workers, Python, Click House, Redis, and other modern technologies to enable seamless data ingestion, transformation, merging, and storage.

Bonus: If you have Web Data Analytics or Programmatic Advertising knowledge, it will be a big plus!.

Responsibilities :

Real-Time Data Processing & Transformation :

- Build low-latency, high-throughput real-time pipelines using Kafka, Redis, Firehose, Lambda, and Cloudflare Workers.

- Perform real-time data transformations like filtering, aggregation, enrichment, and deduplication using Kafka Streams, Redis Streams, or AWS Lambda.

- Merge data from multiple real-time sources into a single structured dataset for analytics.

Batch Data Processing & Transformation :

- Develop batch ETL/ELT pipelines for processing large-scale structured and unstructured data.

- Perform data transformations, joins, and merging across different sources in Click House, AWS Glue, or Python.

- Optimize data ingestion, transformation, and storage workflows for efficiency and reliability.

Data Pipeline Development & Optimization :

- Design, develop, and maintain scalable, fault-tolerant data pipelines for real-time & batch processing.

- Optimize data workflows to reduce latency, cost, and compute load.

Data Integration & Merging :

- Combine real-time and batch data streams for unified analytics.

- Integrate data from various sources (APIs, databases, event streams, cloud storage).

Cloud Infrastructure & Storage :

- Work with AWS services (S3, EC2, ECS, Lambda, Firehose, RDS, Redshift, ClickHouse) for scalable data processing.

- Implement data lake and warehouse solutions using S3, Redshift, and ClickHouse.

Data Visualization & Reporting :

- Work with Power BI, Tableau, or Grafana to create real-time dashboards and analytical reports.

Web Data Analytics & Programmatic Advertising (Big Plus!) :

- Experience working with web tracking data, user behavior analytics, and digital marketing datasets.

- Knowledge of programmatic advertising, ad impressions, clickstream data, and real-time bidding (RTB) analytics.

Monitoring & Performance Optimization:

- Implement monitoring & logging of data pipelines using AWS CloudWatch, Prometheus, and Grafana.

- Tune Kafka, Click House, and Redis for high performance.

Collaboration & Best Practices :

- Work closely with data analysts, software engineers, and DevOps teams to enhance data accessibility.

- Follow best practices for data governance, security, and compliance.

Must-Have Skills :

Programming : Strong experience in Python and JavaScript.

Real-time Data Processing & Merging : Expertise in Kafka, Redis, Cloudflare Workers, Firehose, Lambda.

Batch Processing & Transformation : Experience with Click House, Python, AWS Glue, SQL-based transformations.

Data Storage & Integration : Experience with MySQL, Click House, Redshift, and S3-based storage.

Cloud Technologies : Hands-on with AWS (S3, EC2, ECS, RDS, Firehose, Click House, Lambda, Redshift).

Visualization & Reporting : Knowledge of Power BI, Tableau, or Grafana.

CI/CD & Infrastructure as Code (IaC) : Familiarity with Terraform, CloudFormation, Git, Docker, and Kubernetes.