18 METRIX IT SOLUTIONS Jobs
Data Engineer - Python/Spark (5-15 yrs)
METRIX IT SOLUTIONS
posted 2d ago
Fixed timing
Key skills for the job
Job Description :
Job title : Data Engineer - Engineer II
Exp : 5-15
Location : Chennai
Skill : Python, GCP, ETL-Big Data, PySpark, API, RDBMS, Oracle)
Primary Skills (Essential) :
- Python proficiency (including experience with data manipulation, libraries like pandas)
PySpark expertise (including hands-on experience with Spark transformations and Optimizations)
- Real-time data ingestion experience (e.g., using Kafka, Spark Streaming)
Batch data ingestion :
- Data aggregation and insights generation (using SQL or Spark)
- Rest API integration for data ingestion
- Kafka topic interaction (producing and consuming messages)
- Oracle RDBMS knowledge
- Data purging, and archiving experience
Secondary Skills (Desirable) :
- Familiarity with Generative AI implementation concepts (e.g., data preparation for model training, prompt engineering)
- Understanding of Data lake architecture and Data warehouse concepts
- Having prior experience in AMEX services like Event Engine, HiPED, Hydra deployment
- Having better knowledge on LUMI and Cornerstone data ingestion
Key Responsibilities :
Data Pipeline Development :
- Design, develop, and maintain scalable, efficient, and high-performance data pipelines for ETL (Extract, Transform, Load) processes.
Data Modeling & Warehousing :
- Develop and optimize data models for business intelligence and analytical purposes.
- Manage and improve data warehousing solutions such as Snowflake, Redshift, or BigQuery.
Data Integration :
- Integrate data from various sources, including APIs, third-party tools, and databases, ensuring seamless access to data for analysis.
Collaboration :
- Work closely with data scientists, analysts, and business stakeholders to understand data requirements and deliver data solutions that meet business needs.
Data Quality & Governance :
- Ensure data quality, consistency, and integrity across all systems.
- Implement data governance policies to ensure accurate, accessible, and secure data.
Performance Tuning & Optimization :
- Identify and address performance bottlenecks within data systems, ensuring optimal performance and cost efficiency.
Functional Areas: Software/Testing/Networking
Read full job descriptionPrepare for Data Engineer roles with real interview advice