Data Engineer - Python/Spark (5-15 yrs)

Job Description :

Job title : Data Engineer - Engineer II

Exp : 5-15

Location : Chennai

Skill : Python, GCP, ETL-Big Data, PySpark, API, RDBMS, Oracle)

Primary Skills (Essential) :

- Python proficiency (including experience with data manipulation, libraries like pandas)

PySpark expertise (including hands-on experience with Spark transformations and Optimizations)

- Real-time data ingestion experience (e.g., using Kafka, Spark Streaming)

Batch data ingestion :

- Data aggregation and insights generation (using SQL or Spark)

- Rest API integration for data ingestion

- Kafka topic interaction (producing and consuming messages)

- Oracle RDBMS knowledge

- Data purging, and archiving experience

Secondary Skills (Desirable) :

- Familiarity with Generative AI implementation concepts (e.g., data preparation for model training, prompt engineering)

- Understanding of Data lake architecture and Data warehouse concepts

- Having prior experience in AMEX services like Event Engine, HiPED, Hydra deployment

- Having better knowledge on LUMI and Cornerstone data ingestion

Key Responsibilities :

Data Pipeline Development :

- Design, develop, and maintain scalable, efficient, and high-performance data pipelines for ETL (Extract, Transform, Load) processes.

Data Modeling & Warehousing :

- Develop and optimize data models for business intelligence and analytical purposes.

- Manage and improve data warehousing solutions such as Snowflake, Redshift, or BigQuery.

Data Integration :

- Integrate data from various sources, including APIs, third-party tools, and databases, ensuring seamless access to data for analysis.

Collaboration :

- Work closely with data scientists, analysts, and business stakeholders to understand data requirements and deliver data solutions that meet business needs.

Data Quality & Governance :

- Ensure data quality, consistency, and integrity across all systems.

- Implement data governance policies to ensure accurate, accessible, and secure data.

Performance Tuning & Optimization :

- Identify and address performance bottlenecks within data systems, ensuring optimal performance and cost efficiency.