Lead Data Engineer - SQL/Python (5-7 yrs)

RESPONSIBILITIES :

- Design, build and test batch and streaming data pipelines.

- Build efficient code to transform raw data into datasets for analysis, reporting and machine learning models.

- Work with data scientists to deploy machine learning model outputs into CEPs for recommendations and personalizations.

- Collaborate with other data engineers, data scientists and product managers to implement a shared technical vision.

- Participate in the entire software development lifecycle, from concept to release.

- Contribute to shared data engineering tooling & standards to improve productivity.

- Support, monitor and optimize current and future data infrastructure and platform.

- Provide technical guidance and mentorship to junior engineers.

- Evaluate technologies and conduct proof-of-concepts.

MINIMUM QUALIFICATIONS :

- Applicants should have a degree (B.or higher) in Computer Science or a related discipline or relevant professional experience.

- 5+ years of software development experience designing scalable & automated software systems.

- Experience in processing structured and unstructured data into a form suitable for analysis and reporting.

- Experience with data modelling, batch data pipeline design and implementation.

- Experience building batch or real-time data pipelines.

- Strong software development skills with high proficiency in Python/PySpark or Scala.

- High Proficiency in SQL.

- Experience with data processing frameworks such as Apache Spark (we use Databricks).

- Experience with data transformation tools such as dbt Cloud.

- Experience in cloud-based infrastructures such as AWS or GCP.

- Exposure to orchestration platforms such as Airflow (we use Astronomer).

- Proven attention to detail, critical thinking, and the ability to work independently within a cross-functional team.

- Comfortable with CI/CD (we use GitHub Actions) Pipelines.

- Experience with Git version control, and other software adjacent tools.

- Terraform used as Infra as service tool.