Lead Data Engineer - Big Data (5-9 yrs)

Requirements :

- 6+ years' experience with Big Data Technologies such as Apache Spark, Cloud Native, and Data Lakes, and Databricks.

- Delivered ETL/ELT solutions including data extraction, transformation, cleansing, data integration, and data management.

- Implemented batch ingestion pipelines.

- Strong Python skills - automate tasks, write unit tests using frameworks like Nutter, Unittest, and Pytest.

- Experience with version control systems, such as Git and CI/CD processes.

- Strong knowledge of SQL, including complex queries and performance tuning

- Familiarity with data modeling, data warehousing, and data integration concepts and best practices.

- Documentation and white-boarding skills.

- Strong problem-solving skills.

- Proven ability to work with large cross-functional teams with good communication skills.

Good to have :

- Experience in the retail domain - Aware of entities such as POS (Transaction), Products, Customers, and Inventory.

- Experience with the Build-Operate-Transfer (BOT) model.

- Familiarity with the Azure cloud platform.

- Experience with data visualization tools, such as Power BI.

- Experience with near real-time data processing.

- Experience in Cloud migration methodologies and processes.

- Azure/AWS/Databricks Certified Data Engineer.