Senior Data Engineer - ETL/PySpark (2-4 yrs)

Salary 12-14 LPA

Job Title : Data Engineer

Location : Mumbai, Andheri East

Job Type : Full-time

Experience Required : 2+ years

Job Description :

We are looking for a skilled and passionate Data Engineer with over 2 years of experience to join our team. The ideal candidate will have strong technical expertise in building, optimizing, and maintaining scalable data pipelines and integrations. You will work closely with cross-functional teams to build and enhance data solutions, ensuring seamless data flow across systems while adhering to best practices in performance, security, and scalability.

Key Responsibilities :

- Design, develop, and maintain data pipelines using PySpark for large-scale data processing.

- Perform API Integration to connect various systems, ensuring smooth data exchange and real-time data flow.

- Develop and maintain reliable ETL processes to ensure the accuracy, consistency, and quality of data.

- Implement Kafka for real-time streaming and data ingestion to handle large volumes of data.

- Work with databases, particularly MySQL, to store, manage, and optimize data storage solutions.

- Utilize AWS cloud infrastructure for deploying and managing scalable data solutions.

- Develop and maintain web applications and services using Django and Flask for data management and integration purposes.

- Collaborate with the DevOps team to implement CI/CD pipelines using GitHub for version control, testing, and deployment.

- Monitor data systems for performance, troubleshoot issues and implement best practices for data security and governance.

- Collaborate with other engineers, data scientists, and product teams to define and deliver data-driven solutions.

- Develop and manage Web Crawlers to gather structured data from various web sources for integration into internal systems.

- Design and implement Mobile Crawlers to extract data from mobile apps and APIs, ensuring seamless integration with backend systems.

Required Skills & Competencies :

Programming Languages : Strong proficiency in Python for data engineering and automation tasks.

API Integration : Experience integrating third-party APIs and building APIs for data exchange.

Big Data Technologies : Hands-on experience with PySpark for distributed data processing and Kafka for real-time data streaming.

Databases : Proficiency in MySQL for relational database management, including writing complex queries and optimizing performance.

Web Frameworks :

- Experience with Django and Flask for building web applications and RESTful services.

Version Control :

- Proficient in using GitHub for source code management and collaboration.

Cloud Platforms :

- Knowledge of AWS services like S3, EC2, RDS, Lambda, etc., for data storage, processing, and infrastructure management.

Data Governance & Security :

- Familiarity with best practices in data security, compliance, and governance.

Problem-solving :

- Ability to diagnose and solve complex problems in real time, ensuring data integrity and system efficiency.

Collaboration & Communication :

- Strong interpersonal skills to work collaboratively with teams and communicate technical concepts.

Web Crawler :

- Experience in developing and maintaining web crawlers for data extraction and integration purposes.

Mobile Crawler :

- Knowledge of building mobile crawlers to extract data from mobile applications and environments.

Preferred Qualifications :

- Experience working with distributed systems and handling large datasets.

- Knowledge of containerization technologies like Docker is a plus.

- Familiarity with CI/CD pipelines for automated testing and deployment.

- Experience in optimizing data storage solutions for scalability and performance.

Education :

- Bachelor in technology