TransOrg Analytics - Data Engineer - ETL/SQL (3-5 yrs)

Responsibilities :

1. Design, develop, and maintain robust data pipelines using Azure Data Factory and Databricks workflows.

2. Implement and manage big data solutions using Azure Databricks.

3. Design and maintain relational databases using Azure Delta Lake.

4. Ensure data quality and integrity through rigorous testing and validation.

5. Monitor and troubleshoot data pipelines and workflows to ensure seamless operation.

6. Implement data security and compliance measures in line with industry standards.

7. Continuously improve data infrastructure (including CI/CD) for scalability and performance.

8. Design, develop, and maintain ETL processes to extract, transform, and load data from various sources into Snowflake.

9. Utilize ETL tools (e.g., ADF, Talend) to automate and manage data workflows.

10. Develop and maintain CI/CD pipelines using GitHub and Jenkins for automated deployment of data models and ETL processes.

11. Monitor and troubleshoot pipeline issues to ensure smooth deployment and integration.

12. Design and implement scalable and efficient data models in Snowflake.

13. Optimize data structures for performance and storage efficiency.

14. Collaborate with stakeholders to understand data requirements and ensure data integrity

15. Integrate multiple data sources to create data lake/data mart Perform data ingestion and ETL processes using SQL, Scoop, Spark or Hive

16. Monitor job performances, manage file system/disk-space, cluster & database connectivity, log files, manage backup/security and troubleshoot various user issues

17. Design, implement, test and document performance benchmarking strategy for platforms as well as for different use cases

18. Setup, administer, monitor, tune, optimize and govern large scale implementations

19. Drive customer communication during critical events and participate/lead various operational improvement initiatives