Big Data Engineer

Position Overview: We are seeking a highly skilled Senior Big Data and Container Orchestration Engineer to join our team. In this role, you will be responsible for designing, deploying, and managing container orchestration platforms, as well as setting up and optimizing our big data ecosystem. The ideal candidate will have a proven track record of managing high availability, scalability, and security in a containerized environment, along with deep expertise in big data tools and technologies.

Key Responsibilities:

• Container Orchestration Management:
  
• Design, deploy, and manage container orchestration platforms (SUSE Rancher, Kubernetes, or Talos Linux).
• Ensure high availability, scalability, and security of the containerized environment.

• Big Data Ecosystem Setup:
  
• Install and configure Hadoop, including HDFS, YARN, and MapReduce.
• Deploy and optimize Apache Spark, Kylin, and Druid.
• Ensure seamless integration between Hadoop components and other big data tools.

• Log Management and Kafka Integration:
  
• Configure Kafka clusters to collect and process logs from servers using rsyslog.
• Monitor and optimize Kafka performance.

• Automation and DevOps:
  
• Automate deployment and configuration using tools like Ansible, Terraform, and CI/CD pipelines.
• Implement monitoring and alerting systems.

Requirements:

• Proven experience with SUSE Rancher, Kubernetes, or Talos Linux.
• Deep knowledge of Hadoop, Spark, Kylin, Druid, and Kafka.
• Strong scripting skills in Bash, Python, and SQL.
• Minimum of 5+ years of experience, with at least 2 years in a leadership role.

Qualifications:

• Ability to design and manage container orchestration platforms.
• Expertise in setting up and optimizing a big data ecosystem.
• Proficiency in log management and Kafka integration.
• Experience with automation and DevOps practices.