Data Engineer

About Position:

As a Data Engineer specializing in Spark and Scala, you will be responsible for building, optimizing, and maintaining data pipelines that support both operational and analytical workflows. Your work will directly contribute to ensuring data is ingested, processed, and made available to key stakeholders in an efficient, scalable, and reliable manner.

• Role: Data Engineer
• Location: All PSL Locations
• Experience: 5 To 10 Years
• Job Type: Full Time Employment

What You'll Do:

• Develop and maintain data pipelines using Apache Spark and Scala to process large volumes of data efficiently
• Ability to write and optimize code in Scala for Spark applications.
• Designing, creating, and maintaining Scala-based applications.
• Create real-time data pipelines using Apache Spark Build and maintain SQL databases using Spark SQL Build and maintain a codebase for Apache Spark data processes
• Develop distributed code using the Scala programming language Flow Control in Scala, Functions in Scala, Data Structures in Scala Build, deploy, and run Spark scripts Transform structured data using SparkSQL, DataSets, and DataFrames
• Optimize Spark jobs through partitioning, caching, and other techniques Process continual streams of data with Spark Streaming
• Should have understanding on Spark SQL, DataSets, Running Spark on Clusters, Scala Libraries
• Experience with streaming data architectures and technologies for real-time and low-latency data processing.
• Certification is mandatory

Expertise You'll Bring:

• Spark-Based Data Pipeline Development: Design and implement high-performance data pipelines using Apache Spark and Scala to process large volumes of structured and unstructured data in real-time or batch mode. Focus on optimizing these pipelines for scalability and performance.
• ETL Process Automation: Build, optimize, and maintain automated ETL (Extract, Transform, Load) processes using Spark, ensuring reliable data extraction, transformation, and loading into data storage systems (e.g., data lakes, data warehouses).
• Data Engineering with Scala: Develop robust, reusable, and efficient Scala-based code to integrate Spark with various data sources and sinks. Ensure code is modular, maintainable, and adheres to best practices.
• Performance Optimization: Continuously monitor Spark job performance, identify bottlenecks, and implement optimizations to improve processing speed and reduce resource usage. Leverage Sparks native features (e.g., caching, partitioning) to enhance performance.
• Collaboration with Cross-Functional Teams: Work closely with data scientists, analysts, and business stakeholders to understand data requirements and deliver optimized, reliable data systems that meet organizational needs.
• Data Quality Assurance: Implement robust data validation and error-handling mechanisms to ensure that data processed through Spark pipelines is accurate, clean, and consistent. Ensure data pipelines are resilient and handle data failures gracefully.
• Scalable Architecture Design: Design and implement scalable and distributed systems using Apache Spark that can efficiently process petabytes of data. Ensure systems are highly available, fault-tolerant, and resilient to system failures.
• Cloud Platform Integration: Leverage cloud platforms such as AWS, Google Cloud, or Azure to deploy Spark clusters and manage large-scale data processing. Experience with cloud-native data services (e.g., S3, Redshift, BigQuery) is beneficial.
• Documentation and Best Practices: Write clear, comprehensive documentation for the data pipelines and processes you develop. Establish and maintain coding standards, conduct code reviews, and mentor junior engineers on Spark and Scala best practices.
• Continuous Improvement & Innovation: Stay current with the latest trends in big data technologies, Spark, and Scala development. Propose and implement new approaches to enhance data pipeline efficiency, performance, and capabilities.

Benefits:

• Competitive salary and benefits package
• Culture focused on talent development with quarterly promotion cycles and company-sponsored higher education and certifications
• Opportunity to work with cutting-edge technologies
• Employee engagement initiatives such as project parties, flexible work hours, and Long Service awards
• Annual health check-ups
• Insurance coverage: group term life, personal accident, and Mediclaim hospitalization for self, spouse, two children, and parents

Our company fosters a values-driven and people-centric work environment that enables our employees to:

• Accelerate growth, both professionally and personally
• Impact the world in powerful, positive ways, using the latest technologies
• Enjoy collaborative innovation, with diversity and work-life wellbeing at the core
• Unlock global opportunities to work and learn with the industry’s best

Let’s unleash your full potential at Persistent

“Persistent is an Equal Opportunity Employer and prohibits discrimination and harassment of any kind.”