KreditBee - Data Engineer - Python/Big Data (3-5 yrs)

Key Responsibilities :

Data Product Development : Development of data products such as data warehouses, data lakes, and databases to enhance the capabilities of the Data & Analytics teams.

Technical Leadership : Guide the technical direction for data engineering initiatives, ensuring adherence to best practices in design, implementation, and optimization.

Cross- functional Collaboration : Partner closely with teams across the organization to understand their data needs and design solutions that deliver impactful business outcomes.

Pipeline Optimization & Troubleshooting : Identify and address inefficiencies in data pipelines, optimizing for cost and performance. Troubleshoot and resolve complex issues in the data engineering systems.

Automation & CI/CD : Design, develop, and maintain automated data pipelines and CI/CD processes to enable seamless deployment of new features and fixes.

Skills & Qualifications :

Experience :

- 4+ years of experience in data engineering, particularly in cloud environments and big data technologies.

- Proven experience in designing and building scalable, high- performance data architectures, with familiarity with open- source tools and frameworks.

Technical Skills :

SQL Expertise : Proficient in SQL (preferably MySQL) for querying, managing databases, handling complex queries, and optimising performance.

Programming Proficiency : Strong programming skills, particularly with Python.

Data Pipeline & Architecture : Solid experience in designing and managing data warehouses, data pipelines (ETL/ELT), and data modelling.

Cloud Platforms (AWS) : Hands- on experience with AWS services such as EC2, S3, Lambda, Athena, Kinesis, Redshift, Glue, EMR, DynamoDB, IAM, KMS, Step Functions, SQS, SNS, and CloudWatch.

Real- time & Batch Data Processing : Experience building and maintaining data pipelines for both real- time and batch processing using tools like Spark, Flink, or Kafka.

Version Control : Expertise with Git, GitLab, or GitHub for managing code and collaboration.

Soft Skills :

- Strong analytical and logical thinking with excellent communication skills to collaborate effectively with cross- functional teams.

- Ability to manage multiple priorities while consistently delivering high- quality results on time.

- Adaptability to changing technologies and requirements in a fast- paced work environment.