Data Engineer - PySpark/Azure (3-5 yrs)

We are looking for a talented Data Engineer to join our growing team. The ideal candidate will have experience working with large datasets, designing and implementing robust data pipelines, and leveraging modern data engineering technologies like PySpark, Azure, Azure Data Factory (ADF), Databricks, ETL processes, and SQL. You will be responsible for developing scalable data pipelines, ensuring high data quality, and enabling data-driven decision-making by building optimized solutions on cloud platforms like Azure.

Key Responsibilities :

- Data Pipeline Development : Design, implement, and maintain efficient and scalable ETL pipelines using PySpark, Azure Data Factory (ADF), and Databricks to ingest, transform, and load data from various sources into data storage solutions.

- Data Integration : Integrate data from diverse sources (databases, APIs, flat files, etc.) into a unified data warehouse or lake using SQL, PySpark, and ADF.

- Optimization : Optimize data pipelines and queries to ensure high performance and scalability. Monitor and troubleshoot data pipeline performance issues.

- Cloud Data Platforms : Work with Azure cloud technologies, including Azure Data Lake, Azure SQL Database, Azure Blob Storage, and Azure Synapse Analytics to build and maintain scalable, cloud-based data architectures.

- Collaboration : Collaborate with data scientists, analysts, and other engineers to ensure data is properly structured and available for analysis, modeling, and reporting.

- Data Modeling & Transformation : Design and implement data models, including the transformation and aggregation of raw data into actionable insights.

- Data Quality : Implement data validation, quality checks, and monitoring mechanisms to ensure the accuracy, integrity, and completeness of the data.

- Automation : Automate repetitive data processes and ensure the smooth flow of data between systems with minimal manual intervention.

- Documentation & Reporting : Create clear documentation for data pipelines, systems, and processes. Provide regular reports on the performance, reliability, and status of data workflows.

- Continuous Improvement : Keep up-to-date with industry trends, new tools, and best practices in data engineering. Propose and implement improvements to the existing data infrastructure.