Data Engineer

Key Responsibilities:

ETL Development and Maintenance:

• Design, develop, and maintain scalable ETL processes to extract, transform, and load data from various sources into data warehouses and data lakes.
• Utilize Talend and Azure Data Factory to create and manage ETL workflows that ensure data accuracy and efficiency.

Database Administration:

• Administer and optimize databases, ensuring high performance, availability, and security of data assets.
• Manage Snowflake data warehouse, including schema design, data partitioning, and performance tuning.

Data Engineering and Scripting:

• Develop and maintain Python scripts for data processing, automation, and integration tasks.
• Implement and optimize data transformation processes using DBT (Data Build Tool) to ensure data quality and consistency.

Data Integration and Management:

• Integrate data from multiple sources and ensure seamless data flow and accessibility.
• Collaborate with data analysts and data scientists to understand data requirements and deliver relevant solutions.

Performance Optimization:

• Monitor and optimize data pipelines and database performance to ensure efficient data processing and retrieval.
• Troubleshoot and resolve issues related to data pipelines, database operations, and data quality.

Documentation and Best Practices:

• Document data engineering processes, workflows, and architecture to support knowledge sharing and maintenance.
• Implement best practices for data engineering, including data governance, version control, and code review.

Collaboration and Communication:

• Work closely with cross-functional teams to gather requirements, provide updates, and ensure alignment on data engineering projects.
• Communicate technical concepts and solutions effectively to both technical and non-technical stakeholders.