TMRW - Data Engineer II - ETL (3-5 yrs)

Responsibilities :

- Create, implement and operate the strategy for robust and scalable data pipelines for business intelligence and machine learning.

- Develop and maintain core data framework and key infrastructures

- Create and support the ETL pipeline to get the data flowing correctly from the existing and new sources to our data warehouse.

- Data Warehouse design and data modeling for efficient and cost-effective reporting

- Collaborate with data analysts, data scientists, and other data consumers within the business to manage the data warehouse table structure and optimize it for reporting.

- Constantly striving to improve software development process and team productivity

- Define and implement Data Governance processes related to data discovery, lineage, access control and quality assurance

- Perform code reviews and QA data imported by various processes

Qualifications :

- 3 to 5 years of experience.

- At least 2+ years of experience in data engineering and data infrastructure space on any of the big data technologies: Hive, Spark, Pyspark(Batch and Streaming), Airflow, Redshift and Delta Lake.

- Experience in product-based companies or startups.

- Strong understanding of data warehousing concepts and the data ecosystem.

- Strong Design/Architecture experience architecting, developing, and maintaining solutions in AWS.

- Experience building data pipelines and managing the pipelines after they're deployed.

- Experience with building data pipelines from business applications using APIs.

- Previous experience in Databricks is a big plus.

- Understanding of Dev Ops would be preferable though not a must

- Working knowledge of BI Tools like Metabase, and Power BI is a plus

- Experience of architecting systems for data access is a major plus