Data Engineer

Project Role :Data Engineer


Project Role Description :Design, develop and maintain data solutions for data generation, collection, and processing. Create data pipelines, ensure data quality, and implement ETL (extract, transform and load) processes to migrate and deploy data across systems.

Must have skills :Apache Spark


Good to have skills :Python (Programming Language), Scala, PySpark
Minimum
3 year(s) of experience is required


Educational Qualification :15 years full time education

Summary:We are seeking a highly skilled Data Engineer with 5-10 years of experience to join our dynamic team. The ideal candidate will have extensive experience with Hadoop CDP and a strong background in data engineering.
Roles & Responsibilities:-Technical Leadership:Lead and mentor a team of developers in building scalable and efficient data processing solutions using Apache Spark and Scala.Code Quality & Best Practices:Ensure the delivery of high-quality code that adheres to best practices in Spark and Scala development, such as modularity, scalability, and reusability.Performance Optimization:Optimize Spark applications for high performance, scalability, and stability. Identify and address bottlenecks and inefficiencies.Collaboration:Work closely with cross-functional teams, including data engineers, architects, and product owners, to understand requirements and deliver solutions that meet business objectives.Data Engineering:Collaborate in the development of data solutions that involve ETL/ELT processes, streaming data pipelines, and batch data processing.Troubleshooting & Support:Provide advanced troubleshooting, root cause analysis, and support for Spark applications in production environments.1. Develop and Maintain Data Pipelines:Design, develop, and maintain scalable and efficient data pipelines using Hadoop CDP.Optimize and tune data pipelines for performance and scalability.2. Data Processing and Analysis:Implement data processing workflows using Hadoop CDP.Perform data analysis and generate insights to support business decisions.3. Data Integration:Integrate data from various sources into the Hadoop ecosystem.Ensure data quality and consistency across different data sources
Professional & Technical Skills:

Experience:Minimum of 7+ years of experience in big data technologies, with a strong focus on Apache Spark and Scala.Proficiency:Deep expertise in Apache Spark (including Spark Core, Spark SQL, and Spark Streaming) and Scala programming.Data Systems:Strong experience with distributed data processing frameworks, data lakes, data warehouses, and cloud-based data platforms (e.g., AWS, Azure, or Google Cloud).Qualifications 
15 years full time education