Data Engineer - Scala/Python (1-4 yrs)

Job Description :

Responsibilities :

- Develop ETL (Extract, Transform, Load) processes to move and transform data between systems.

- Integrate data from various sources, ensuring consistency and accuracy.

- Monitor system performance and identify areas for optimization.

- Align architecture with business requirements and provide solutions which fit best to solve the business problems.

- Build the infrastructure required for optimal ETL pipelines from a wide variety of data sources using SQL and AWS - big data- technologies.

- Evaluate and recommend new technologies to enhance data processing capabilities.

- Work closely with data scientists, analysts, and other stakeholders to understand their data needs.

- Use programming language( python, spark) and tools efficiently to collate the data

- Identify ways to improve data reliability, efficiency and quality.

- Use data to discover tasks that can be automated.

Requirements :

- Can Develop ETL (Extract, Transform, Load) processes to move and transform data between systems.

- Have good hands-on experience in any coding language Python or Scala.

- Good understanding of Data Engineering concepts such as data warehousing, datalakes, data modeling, data quality assurance, data lifecycle management, streaming data, metadata management etc.

- Strong hold on cloud-based ecosystems(AWS preferred).

- Have an understanding of data pipeline orchestration tools such as Airflow or Azkaban (Airflow Preferable).

- Can work with spark, EMR and Kubernetes cluster.

- Can write spark jobs, shell scripts and optimized data pipelines.

- Experience with relational SQL and NoSQL databases, including Postgres, Amazon Redshift and MongoDB.

- Have an understanding of OOPS-based data platforms, and can develop data platform modules.