Data Engineer - ETL/Data Pipeline (3-8 yrs)

Key Responsibilities :

- Design and implement scalable and reliable ETL/ELT pipelines to ingest, transform, and load data from various sources into data warehouses and data lakes.

- Develop and maintain data integration and ingestion processes, ensuring data accuracy and consistency.

- Optimize data pipelines for performance, efficiency, and reliability.

- Implement data quality checks and monitoring to ensure data integrity.

- Architect and deploy data solutions on either AWS or Azure, demonstrating mastery of the platform's data services.

- Utilize cloud-native tools and services for data storage, processing, and analytics.

- Manage and optimize cloud infrastructure for data workloads.

- Design and implement effective data models (dimensional, relational, etc.) to support business requirements.

- Ensure data models are scalable, maintainable, and aligned with best practices.

- Develop and implement data transformation logic using scripting languages (Python, PySpark, SQL).

- Perform data cleansing, validation, and enrichment.

- Optimize data processing for large datasets.

- Utilize Git (GitHub) for version control and collaborative development.

- Implement CI/CD pipelines using GitHub Actions or Azure DevOps for automated deployments.

- Employ testing frameworks (PyTest) and code quality tools (SonarQube) to ensure code reliability.

- Work with various data platforms and tools, including:

- Data Factory (Azure) or Glue (AWS) for ETL.

- Databricks for Spark-based data processing.

- SQL Databases (Azure SQL DB, Redshift) for structured data storage.

- Synapse Analytics (Azure) for data warehousing and analytics.

- Stream Analytics (Azure) or Kinesis (AWS) for real-time data processing.

- Airflow for workflow orchestration.

- Identify and resolve data-related issues, including performance bottlenecks, data quality problems, and pipeline failures.

- Troubleshoot and debug complex data processing workflows.

Required Skills and Qualifications :

- AWS or Azure cloud platforms.

- ETL/ELT concepts and implementation.

- Data modeling principles and techniques.

- Data integration and ingestion.

- Data manipulation and processing.

- Data Factory, Databricks, SQL Databases, Synapse, Stream Analytics, Glue, Airflow, Kinesis, Redshift.

- Github, Github Actions, Azure Devops.

- SonarQube, and PyTest.

- Python.

- PySpark/Spark-SQL.

- SQL.

- Linux environments.

Experience :

- Proven experience in building and maintaining large-scale data pipelines.

- Experience working with diverse data sources and formats.

- Experience with both batch and real-time data processing.

- Experience with building Data lakes and Data Warehouses.

Education :

- A Master's degree in Computer Science, Data Science, or a related field is preferred.

Soft Skills :

- Strong problem-solving and analytical skills.

- Excellent communication and collaboration skills.

- Ability to work independently and as part of a team.

- Strong attention to detail