Big Data Architect - SQL/Python (8-13 yrs)

Job Description

We are seeking a highly skilled Senior Big Data Engineer with expertise in designing, optimizing, and managing large-scale Big Data systems.

The ideal candidate will possess a deep understanding of Big Data platforms, cloud technologies, and DevOps practices, along with strong leadership and advisory skills to guide customers on strategic initiatives.

This role requires hands-on experience with industry-standard tools, programming languages, and Big Data frameworks.

Key Responsibilities :

- Big Data System Design and Optimization Architect, develop, and optimize Big Data systems on Cloudera platforms (cloud or on-premise).

- Design Hadoop clusters, including capacity planning, performance tuning, and data ingestion strategies.

- Evaluate and implement scalable data pipeline solutions for batch and real-time processing.

- Provide architectural guidance on data modeling, storage, and partitioning for optimal performance.

- Data Migration and Validation Develop comprehensive data and metadata migration plans, ensuring high accuracy and efficiency.

- Design and implement data validation and reconciliation methods.

- Identify migration dependencies, risks, and success criteria, and devise mitigation strategies.

- Cloud and DevOps Enablement Leverage expertise in cloud platforms (AWS, Azure, or Google Cloud) for scalable Big Data solutions.

- Implement DevOps practices, including Git, CI/CD pipelines, and Infrastructure as Code (IaC).

- Automate workflows and optimize resource utilization to reduce operational costs.

- Big Data Toolset Expertise Proficiently utilize Apache Spark, Hive, Impala, and Kafka for data processing and streaming.

- Hands-on experience with HBase, Apache Storm, Knox, Ozone, Iceberg, and Ambari.

- Develop and manage data pipelines using Python, Scala, and relevant scripting languages.

- Monitor, debug, and optimize distributed systems to ensure reliability and scalability.

- Enterprise Security and Compliance Integrate enterprise security solutions such as LDAP, Kerberos, and role-based access controls.

- Implement data encryption, masking, and secure access protocols to meet compliance standards.

- Ensure adherence to security frameworks like GDPR, HIPAA, or other industry regulations.

- Collaboration and Mentoring Collaborate with cross-functional teams to align on project goals and deliverables.

- Mentor and upskill team members, fostering a culture of continuous learning.

- Conduct workshops and training sessions to share knowledge and best practices.

- Support and Troubleshooting Provide ongoing support for "Run The Engine" activities, ensuring system reliability.

- Troubleshoot and resolve performance bottlenecks, failures, and scalability challenges.

- Conduct root cause analysis for production incidents and implement preventive measures.

Required Skills and Qualifications :

Core Technical Expertise Big Data Frameworks :

- Apache Spark, Hive, Impala, Kafka, HBase.

Programming Languages :

- Python, Scala, Java, Bash, or Perl.

Cloud Platforms :

- AWS, Azure, Google Cloud Platform (GCP).

Data Management :

- Expertise in Hadoop Distributed File System (HDFS), Apache Iceberg, and NoSQL databases.

Security Solutions :

- LDAP, Kerberos, encryption techniques, and access control mechanisms.

- DevOps and Automation Skills Proficiency with Git, CI/CD tools (Jenkins, GitLab, CircleCI), and Infrastructure as Code (Terraform, CloudFormation).

- Automation experience using Ansible, Chef, Puppet, or similar tools.

- Exposure to containerization and orchestration tools like Docker and Kubernetes.

- Analytical and Problem-Solving Skills Strong ability to design efficient data models and optimize large-scale data pipelines.

- Exceptional problem-solving skills to debug and resolve complex technical issues.

- Proficiency in monitoring and logging tools such as Grafana, Prometheus, or Splunk.

- Preferred Skills Knowledge of data lake architectures and advanced analytics solutions.

- Experience with real-time data streaming frameworks like Flink or Beam.

- Familiarity with machine learning workflows and tools (TensorFlow, PyTorch, or Spark MLlib).

- Experience working with international clients in Europe or the US region