i
Careernet
22 Careernet Jobs
Principal Engineer - Python/Spark (10-15 yrs)
Careernet
posted 2d ago
Fixed timing
Job Description :
Requirements :
- Strong experience (12+ years) in Scala programming language and extensive experience with Apache Spark for Big Data processing for design, developing and maintaining scalable on-prem and cloud environments, especially on AWS EMR.
- Proficiency in performance tuning of Spark jobs, optimizing resource usage, shuffling, partitioning, and caching for maximum efficiency in Big Data environments.
- In-depth understanding of the Hadoop ecosystem, including HDFS, YARN, and MapReduce.
- Expertise in designing and implementing scalable, fault-tolerant data pipelines with end-to-end monitoring and alerting.
- Using Python to develop infrastructure modules. Hence, hands-on experience with Python.
- Solid grasp of database systems and SQLs for writing efficient SQLs (RDBMS/Warehouse) to handle TBS of data.
- Familiarity with design patterns and best practices for efficient data modeling, partitioning
strategies, and sharding for distributed systems and experience in building, scheduling, and maintaining DAG workflows.
- End-to-end ownership with definition, development, and documentation of software's objectives, business requirements, deliverables, and specifications in collaboration with stakeholders.
- Experience in working on GIT (or equivalent source control) and solid understanding of Unit and integration test frameworks.
- Must have the ability to collaborate with stakeholders/teams to understand requirements and develop a working solution and the ability to work within tight deadlines and effectively
prioritize and execute tasks in a high-pressure environment.
- Must be able to mentor junior staff.
Functional Areas: Other
Read full job descriptionPrepare for Principal Engineer roles with real interview advice