143 Big Data Engineer Jobs

We are looking for a highly skilled Big Data Engineer with deep expertise in Apache Nifi, Zookeeper, Apache Kafka, Apache Spark, and Scala programming. The ideal candidate should have strong SQL knowledge with PostgreSQL and hands-on experience with AWS services such as EMR, ElasticSearch, S3, and RDS (PostgreSQL). You will be responsible for designing, developing, and optimizing scalable data pipelines, ensuring data integrity, and troubleshooting AWS cloud infrastructure issues.

• Design, implement, and maintain data pipelines using Apache Nifi.
• Manage and configure Zookeeper to support distributed systems.
• Develop real-time and batch processing applications using Apache Kafka and Apache Spark.
• Write efficient and optimized Scala code for data processing.
• Perform data ingestion, transformation, and storage in a distributed environment.

• Work with PostgreSQL, writing complex SQL queries for data extraction, transformation, and analysis.
• Optimize database performance and ensure data consistency.
• Implement database backup, replication, and recovery strategies.

• Deploy and manage AWS EMR for big data processing.
• Utilize AWS ElasticSearch for indexing and searching large datasets.
• Store and manage data in AWS S3 for scalable storage solutions.
• Work with AWS RDS (PostgreSQL) for database management and operations.
• Troubleshoot AWS services and optimize performance.

• Administer and manage Windows and Linux OS (Ubuntu/AWS Linux).
• Develop Shell scripts for automation and workflow management.

• Diagnose and resolve issues related to AWS cloud services, big data applications, and distributed systems.
• Monitor system performance and optimize pipelines for better efficiency.
• Ensure high availability and fault tolerance of data pipelines and services.

• Strong expertise in Apache Nifi, Zookeeper, Apache Kafka, Apache Spark, and Scala programming.
• Excellent SQL skills with PostgreSQL, including query optimization and indexing.
• Hands-on experience with AWS services: AWS EMR, AWS ElasticSearch, AWS S3, AWS RDS (PostgreSQL).
• Proficiency in Windows and Linux (Ubuntu/AWS Linux) administration.
• Strong problem-solving skills and ability to troubleshoot AWS infrastructure and big data applications.
• Experience in Shell scripting for automation.

• AWS certifications (e.g., AWS Certified Big Data - Specialty, AWS Certified Solutions Architect).
• Experience with Terraform or AWS CloudFormation for infrastructure as code (IaC).
• Knowledge of Kubernetes for container orchestration.
• Experience with CI/CD pipelines for data workflows.
• Exposure to monitoring tools such as Prometheus, Grafana, or AWS CloudWatch.