Big Data Architect

We are seeking a highly skilled Big Data Architect with extensive experience in Apache Spark and a deep understanding of big data ecosystems. The ideal candidate will design, build, and manage large-scale distributed data processing systems, ensuring data pipelines are scalable, efficient, and maintainable. You will be responsible for architecting solutions that meet business needs while optimizing for performance and cost.

Key Responsibilities:

• Architect and Design end-to-end big data solutions that handle massive amounts of structured, semi-structured, and unstructured data.
• Lead the development and deployment of distributed data processing systems, particularly with Apache Spark and related big data technologies.
• Collaborate with stakeholders to understand data processing requirements, translating them into scalable data architecture and design.
• Optimize data processing workflows to improve performance, scalability, and cost-effectiveness.
• Ensure data security and governance by implementing best practices and complying with data privacy regulations.
• Integrate diverse data sources, including on-premise and cloud-based systems, into a unified data architecture.
• Mentor and guide junior engineers and data engineers on big data technologies and best practices.
• Stay up-to-date with the latest advancements in big data technologies, especially Spark, Hadoop, etc.
• Troubleshoot and resolve technical challenges in data processing and data pipeline failures.

• Experience in big data architecture, design, and development.
• Hands-on experience with Apache Spark in a production environment, including Spark Core, Spark SQL, and Spark Streaming.
• Strong proficiency in distributed computing and parallel processing .
• Deep understanding of big data technologies, including Hadoop, HDFS, Kafka, Hive, HBase , and other tools in the big data ecosystem.
• Proficient in programming languages such as Scala, Python , or Java , particularly for Spark application development.
• Experience with data modeling, ETL pipelines , and data warehousing.
• Knowledge of database systems such as NoSQL (SOLR, Hbase, Redis) and SQL databases is added advantage
• Strong problem-solving and analytical skills with a focus on scalability and performance optimization.
• Familiarity with containerization tools like Docker and orchestration platforms such as Kubernetes is a plus.