Senior Data Engineer - Java/Python/Scala (5-9 yrs)

Title : Senior Data Engineer

Job Location : Gurgaon

Education : B.TECH / MTECH

Experience : 5 + yrs

Permanent Payroll : Nippon Data Systems Ltd

NIPPON DATA SYSTEMS LIMITED is a provider of IT enabled business solutions which provide our clients competitive business advantage. We partner with our customers to deliver business solutions to their satisfaction and help them 'Live Your Enterprise Potential- '.

Since the time NDS was established in 1994, we have built a solid foundation of success on which we continue to grow. We specialize in solving business problems by harnessing technology and applying them for developing and implementing customized and standard information systems.

Client : Keysight Technologies

Keysight Technologies, or Keysight, is a US company that manufactures electronic test and measurement equipment and software. In 2014, Keysight was spun off from Agilent Technologies, bringing with it the product lines focused on electronics and radio, leaving Agilent with the chemical and bio-analytical products. The name is a blend of key and insight, and was chosen to convey that the company "unlocks" "critical or key insights

Keysight's products include hardware and software for benchtop, modular, and field instruments. Instruments include oscilloscopes, multimeters, logic analyzers, signal generators, spectrum analyzers, vector network analyzers, atomic force microscopes (AFM), automated optical inspection, automated X-ray inspection (5DX), in-circuit testers, power supplies, tunable lasers, optical power meters, wavelength-meters, electro-optic converters, optical modulation analyzers and handheld tools. In addition, it produces electronic design automation (EDA) software (EEsof division). It mainly serves the telecommunications, aerospace/defense, industrial, computer, and semiconductor industries.

Job Overview :

As a Senior Data Engineer, you will play a pivotal role in architecting and implementing our data infrastructure transformation. You will be responsible for converting complex, legacy data into structured, high-quality datasets that fuel AI and machine learning models. This role necessitates a profound comprehension of modern data technologies and a strong enthusiasm for converting diverse datasets into insightful information that drives strategic decision-making.

Key Responsibilities :

- Legacy Data Transformation : Design and execute strategies to migrate and transform legacy data systems into modern, scalable architectures that are optimized for AI and ML workloads.

- Data Architecture & Modeling : Develop and maintain an efficient data architecture that supports real-time and batch processing, ensuring data integrity, security, and compliance.

- Platform Development : Work with modern data stack technologies (e.g., Hadoop, Spark, Kafka, AWS/Azure/GCP cloud services, SQL, NoSQL databases) to build and optimize data pipelines.

- AI/ML Enablement : Collaborate with data scientists and ML engineers to ensure data is properly formatted, cleaned, and enriched for AI model development and deployment.

- Performance Tuning : Optimize data processing pipelines for speed and efficiency, addressing bottlenecks and ensuring scalability.

- Data Governance : Implement and enforce data governance policies, including metadata management, data quality control, and data lineage tracking.

- Innovation & Best Practices : Stay abreast of emerging trends in data engineering and AI, introducing new tools and techniques to improve data infrastructure.

- Team Leadership : Mentor junior team members, fostering a culture of continuous learning and innovation.

Requirements :

- Bachelor's or master's degree in computer science, Data Engineering, or a related field.

- A minimum of 7 years of hands-on experience in data engineering, with a focus on large-scale data migrations and transformations.

- Proficiency in programming languages such as Python, Java, or Scala.

- Extensive experience with big data technologies (Hadoop, Spark, Hive, HBase, etc.) and cloud-based data solutions (AWS S3, Azure Data Lake, GCP BigQuery).

- Strong understanding of database design, ETL processes, and data warehousing concepts.

- Experience with data modeling, data warehousing, and OLAP systems.

- Familiarity with containerization (Docker, Kubernetes) and CI/CD pipelines.

- Excellent problem-solving skills and ability to work collaboratively with cross-functional teams.

- Experience in data security, privacy, and GDPR/CCPA compliance.

- Strong communication skills and the ability to articulate complex technical concepts to non-technical stakeholders