Data Engineer - ETL/Scala/Python (3-10 yrs)

ETL, Python, Pyspark, SQL, GCP - Big query,ADF,AWS -GLue, Excel

Job Summary :

We are looking for a skilled Data Engineer to design, build, and maintain scalable data pipelines and ETL processes. The ideal candidate will have strong experience with AWS, SQL, and BigQuery, ensuring efficient data integration, transformation, and storage.

Key Responsibilities :

- Design, develop, and optimize ETL pipelines to process large-scale data from multiple sources.

- Work with AWS services (S3, Glue, Redshift, Lambda, etc.) to build cloud-based data solutions.

- Develop and manage BigQuery datasets, optimizing queries and storage for performance and cost.

- Write efficient SQL queries for data extraction, transformation, and reporting.

- Collaborate with data analysts, scientists, and business teams to define data requirements.

- Implement data governance, security, and quality best practices.

- Monitor and troubleshoot data pipeline issues, ensuring high availability and reliability.

Required Skills & Experience :

- 2+ years of experience in Data Engineering or a related field.

- Strong knowledge of SQL, BigQuery, and AWS services (S3, Glue, Redshift, Athena, etc.).

- Hands-on experience with ETL tools and data pipeline development.

- Proficiency in Python or Scala for data processing.

- Understanding of data modeling, warehousing, and performance optimization.

- Experience with workflow orchestration tools like Airflow or AWS Step Functions.

- Strong problem-solving and communication skills.

Preferred Qualifications :

- Experience with real-time data processing (Kafka, Kinesis, etc.).

- Knowledge of CI/CD for data pipelines and Infrastructure as Code (IaC).

- Exposure to machine learning or advanced analytics workflows.