Technical Lead - Big Data (8-10 yrs)

Job Description :

Role Overview :

As a Tech Lead - Big Data, you will be responsible for leading data-driven initiatives, designing scalable big data solutions, and ensuring the successful implementation of AWS-based cloud services. This role demands strong expertise in distributed data processing frameworks, OLAP systems, and programming languages like Java and Scala.

Key Responsibilities :

Data Engineering & Solution Design :

- Lead the design, development, and optimization of big data pipelines and workflows.

- Work with AWS Cloud Services to build and manage scalable data platforms.

- Develop solutions leveraging ClickHouse or other OLAP systems for real-time data analytics.

Distributed Systems Development :

- Utilize distributed data processing frameworks like Apache Spark (Core, Streaming, SQL), Storm, or Flink to process large-scale data efficiently.

Programming & Framework Expertise :

- Code and maintain data systems using Java, Scala, and optionally Python.

- Implement robust, reusable, and optimized code to handle complex data challenges.

Cloud Services Management :

- Work with data-related AWS Cloud Services, such as AWS Glue, Redshift, EMR, S3, or Lambda.

- Ensure cloud infrastructure supports scalable, efficient, and cost-effective data solutions.

Leadership & Collaboration :

- Mentor junior engineers and guide the team in adopting best practices for data engineering.

- Collaborate with cross-functional teams to gather requirements and deliver high-quality solutions.

Performance Optimization :

- Optimize big data platforms for speed, scalability, and efficiency.

- Troubleshoot and resolve bottlenecks in data pipelines and workflows.

Required Skills and Qualifications :

Technical Expertise :

- Programming Languages: Proficient in Java and Scala, with good knowledge of Python.

- OLAP Systems: Experience with ClickHouse or equivalent systems.

- Distributed Data Processing Frameworks: Hands-on experience with Apache Spark, Storm, or Flink.

AWS Cloud Services :

- Proficiency in data-related AWS services (e.g., Glue, Redshift, S3, Lambda).

- Familiarity with managing large-scale cloud data platforms.

Data Engineering Tools :

- Expertise in distributed computing, big data analytics, and pipeline automation.

- Strong understanding of data modeling, ETL processes, and performance tuning.

Soft Skills :

- Strong problem-solving abilities and analytical skills.

- Excellent communication and leadership skills.

- Ability to work in a collaborative, fast-paced environment.

Preferred Qualifications :

- Familiarity with real-time analytics systems.

- Experience in implementing solutions for the Digital Business domain.

- Knowledge of Kubernetes, Docker, or other containerization tools.