Big Data Solution Architect

Job Description

We are seeking a highly skilled and experienced Big Data Solution Architect to join our dynamic team. In this role, you will be responsible for designing and implementing cutting-edge big data solutions that address the evolving needs of our organization. You will collaborate with cross-functional teams, including data engineers, data scientists, and business stakeholders, to deliver robust and scalable data solutions.

Key Responsibilities

• Technical and Architectural Expertise: Lead the design and implementation of big data platforms using technologies such as Hadoop, HBase, Hive, Spark, Kafka. Experience in setting up Hadoop clusters from scratch without relying on pre-designed platforms.
• Solution Development: Develop and deploy Hadoop-based solutions for machine learning analytics, data archiving, text analytics, IoT, and reporting.
• Containerized and Private Cloud Environments: Set up and manage containerized environments and private clouds.
• Stakeholder Engagement: Work closely with stakeholders to understand their objectives and requirements, translating them into technical solutions that drive business value.
• Data Governance: Establish and enforce data governance policies to ensure data quality, security, and compliance.
• Team Management: Lead and mentor a team of data engineers and other technical staff, fostering a collaborative and innovative work environment. Experience in managing people and providing technical leadership.
• Collaboration: Collaborate with data engineers to ensure seamless integration and implementation of big data solutions, understanding data flows and the needs of teams working with data.
• Scalability Mindset: Design a platform capable of scaling from a small number of users to millions, ensuring the design is robust and efficient.
• Continuous Improvement: Stay current with the latest trends and advancements in big data technologies, continuously improving and evolving our data infrastructure.

Qualifications

• Education: Bachelors or Masters degree in Computer Science, Engineering, Information Systems, or a related field.
• Experience: 10+ years of extensive experience with big data solutions and architecture, including hands-on experience with Hadoop, HBase, Hive, Spark, Kafka.
• Certifications: Relevant certifications in big data technologies and cloud platforms.
• Project Management: Experience with agile methodologies and project management tools.
• Container Orchestration Management: Design, deploy, and manage container orchestration platforms (SUSE Rancher, Kubernetes, or Talos Linux). Ensure high availability, scalability, and security of the containerized environment.
• Big Data Ecosystem Setup: Install and configure Hadoop, including HDFS, YARN, and MapReduce. Deploy and optimize Apache Spark, Kylin, and Druid. Ensure seamless integration between Hadoop components and other big data tools.
• Log Management and Kafka Integration: Configure Kafka clusters to collect and process logs from servers using rsyslog. Monitor and optimize Kafka performance.
• Automation and DevOps: Automate deployment and configuration using tools like Ansible, Terraform, and CI/CD pipelines. Implement monitoring and alerting systems