Data Engineer - Python (2-5 yrs)

About NonStop io Technologies :

NonStop io Technologies is a value-driven company with a strong focus on process-oriented software engineering.

We specialize in Product Development and have a decade's worth of experience in building web and mobile applications across various domains.

NonStop io Technologies follows core principles that guide its operations and believes in staying invested in a product's vision for the long term.

We are a small but proud group of individuals who believe in the 'givers gain' philosophy and strive to provide value in order to seek value.

We are committed to and specialize in building cutting-edge technology products and serving as trusted technology partners for startups and enterprises.

We pride ourselves on fostering innovation, learning, and community engagement.

Join us to work on impactful projects in a collaborative and vibrant environment.

Brief Description :

We are looking for a talented Data Engineer to join our team. In this role, you will design, implement, and manage data pipelines, ensuring the accessibility and reliability of data for critical business processes.

This is an exciting opportunity to work on scalable solutions that power data-driven decisions

Skillset :

Here is a list of some of the technologies you will work with (the list below is not set in stone)

Data Pipeline Orchestration and Execution :

- AWS Glue

- AWS Step Functions

- Databricks Change

Data Capture :

- Amazon Database Migration Service

- Amazon Managed Streaming for Apache Kafka with Debezium Plugin

Batch :

- AWS step functions (and Glue Jobs)

- Asynchronous queueing of batch job commands with RabbitMQ to various "ETL Jobs"

- Cron and subervisord processing on dedicated job server(s) : Python & PHP

Streaming :

- Real-time processing via AWS MSK (Kafka), Apache Hudi, & Apache Flink

- Near real-time processing via worker (listeners) spread over AWS Lambda, custom server (daemons) written in Python and PHP Symfony

- Languages : Python & PySpark, Unix Shell, PHP Symfony (with Doctrine ORM)

- Monitoring & Reliability : Datadog & Cloudwatch

Things you will do :

- Build dashboards using Datadog and Cloudwatch to ensure system health and user support

- Build schema registries that enable data governance

- Partner with end-users to resolve service disruptions and evangelize our data product offerings

- Vigilantly oversee data quality and alert upstream data producers of issues

- Support and contribute to the data platform architecture strategy, roadmap, and implementation plans to support the company's data-driven initiatives and business objective

- Work with Business Intelligence (BI) consumers to deliver enterprise-wide fact and dimension data product tables to enable data-driven decision-making across the organization.

- Other duties as assigned