Data Engineer - Python/SQL (5-6 yrs)

Job Description :

Roles and Responsibilities :

What You'll Do :

- Lead the technical design and architecture of our analytics platforms in the cloud.

What You'll Do :

- Understand business requirements and translate them into a well-defined technical architecture with a clear data strategy.

- Develop and maintain scalable data pipelines and architectures for our data infrastructure.

- Provide technical mentorship to developers, guiding them on data related patterns and best practices.

- Evaluate and implement cost-effective solutions.

- Stay up to date with industry trends and emerging technologies in big data engineering and AI.

- Implement and maintain data security and privacy standards in compliance with relevant regulations.

- Develop and maintain web APIs to enable data access and integration with other systems.

- Document technical decisions, usage instructions and architecture diagrams for future reference and knowledge sharing.

- Collaborate closely with other members of the data team, including analysts, and developers, to ensure that our data infrastructure is robust, scalable, and optimized for performance.

- Responsible for running POCs to solve business problems and productionize the concepts.

Who You Are :

- Minimum 4+ years of experience building scalable near real time solutions.

- Solid understanding of data pipelines, including near real time processing, streaming and batch.

- Excellent Python skills and experience in building scalable web API services.

- Having .NET skill is an added advantage.

- Should have a very good knowledge of SQL (Postgres Preferred) & NOSQL.

- Should have good knowledge of distributed document DB like Mongo or Casandra or Elasticsearch.

- Previous experience with near real time data ingesting, processing and analysis.

- Experience with Apache Kaftka, Apache Spark, AWS Kinesis, AWS Glue or similar technologies.

- Experience building dashboards in Metabase, PowerBI, AWS Quicksight or similar BI tools.

- Strong problem-solving skills and the ability to think critically to identify and resolve complex technical challenges.

- Excellent communication and collaboration skills to work effectively with designers, developers, product managers, and stakeholders.

- Experience with AWS services.

- Experience with unit testing