Data Engineer

Key Responsibilities:

• Design, develop, and maintain scalable and efficient data pipelines using Azure Databricks
• Optimize and troubleshoot existing data pipelines to enhance performance and reliability.
• Ensure data quality, integrity, and consistency across various data sources.
• Implement ETL processes and manage data flows into Data Warehouses and Data Marts.
• Develop and optimize Snowflake for data processing.
• Proficient in Python for data processing, transformation, and automation tasks.
• Must have experience with writing own unit testing/test cases.
• Monitor pipeline performance, proactively identify issues, and conduct necessary maintenance and updates.
• Maintain comprehensive documentation of data processes, architectures, and technical specifications.

Required Skills:

• Azure Databricks & Apache Spark
• Python
• Data Warehousing & Data Mart
• ETL Development
• GitHub for version control and collaboration
• JIRA for work management
• Knowledge of DBT Framework
• Experience Range: 3 to 5 years

Interpersonal Skills:

• Strong problem-solving and analytical abilities.
• Excellent written and verbal communication skills.
• Ability to work effectively within a team and collaborate.

Good to Have (Preferred Skills):

• Experience with Snowflake for cloud-based data warehousing.