Senior Data Engineer - PySpark/Hadoop (4-10 yrs)

Role : Senior Data Engineer

Experience : 4+ yrs

Work Mode : Remote

Joining Time Frame : Immediate to 15 days

Key Responsibilities :

- Design and architect robust data pipelines for structured, semi-structured, and unstructured data.

- Develop, manage, and optimize databases, including RDMS (MySQL, PostgreSQL), NoSQL (MongoDB), and data lakes (S3).

- Implement efficient ETL processes using tools like PySpark and Hadoop to transform and prepare data for analytics and #AI use cases.

- Optimize database performance, including query tuning, indexing, and caching strategies using tools like Redis and AWS-specific caching databases.

- Build and maintain CI/CD pipelines, manage YML files, and use GitHub for version control and collaboration.

- Leverage Docker for containerized deployment, with hands-on experience in running Docker commands for database and pipeline management.

- Ensure solutions adhere to best practices in system design, focusing on trade-offs, security, performance, and efficiency.

- Monitor, maintain, and troubleshoot database infrastructure to ensure high availability and performance.

- Collaborate with engineering teams to design scalable solutions for large-scale data processing.

- Stay updated on the latest database technologies and implement best practices for database design and management.

Qualifications :

- 4+ years of experience in database architecture and optimization.

- Expertise in RDMS, NoSQL, and semi-structured databases (MySQL, PostgreSQL, MongoDB).

- Proficiency in programming languages for database integration and optimization (Python preferred).

- Strong knowledge of distributed data processing tools like PySpark and Hadoop.

- Hands-on experience with AWS services for data storage and processing, including S3.

- Strong familiarity with Redis for caching and query optimization.

- Proven experience with Docker for containerized deployments and writing CI/CD pipelines using YML files.