Data Engineer

Responsibilities:

1. Design, construct, install, test and maintain highly scalable data management systems.

2. Monitor the system performance by performing regular tests, troubleshooting and integrating new features.

3. Work with large, complex data sets to solve difficult, non-routine analysis problems.

4. Collaborate with cross functional teams to understand data needs, implement data collectin and data storage strategies that optimize efficiency and quality.

5. Interface with various teams (QA, SRE, Architects, etc.). Manage risks and resolves issues that affect release scope, schedule and quality.

6. Participate in the entire software development lifecycle, from requirement analysis to testing and maintenance.

7. Ensure data integrity and compliance with data privacy regulations.

8. Participate in code reviews and contribute to team knowledge sharing.

Qualifications:

1. Bachelor's degree in Computer Science or related field (Master's preferred) with 4-8 years of overall work experience.

2. Expertise in ETL processes, data warehousing and data pipeline construction and workflow management tools (eg Apache Airflow).

3. Experience in developing, managing and debugging ETL Pipelines running on orchestration frameworks like Airflow

4. Proficiency in programming languages such as GoLang, Python, Java etc.

5. Hands-on experience of designing high-volume and scalable systems is a plus.

6. Hands-on experience with cloud service (AWS experience is a plus ) and various cloud data warehousing technologies (snowflake is a plus) and tools and platforms like Docker, Kubernetes, monitoring tools like Grafana is a plus.

7. Exposure to Microservices architecture.

8. Experience with CI-CD practices using GitHub Actions, Concourse etc.

9. Strong analytical and problem-solving skills and ability to learn fast and adopt new technologies into the project.

10. Excellent communication, mentoring and teamwork abilities.