Data Engineer

Job responsibilities:

• Data Architecture & ETL Development: Design & develop robust data architecture. Design & develop scalable ETL design to build performant data pipeline solutions & feature engineering. Write well-structured, efficient, and maintainable code.
• Data Analysis & Feature Engineering: Develop Data Quality checks, perform data analysis and identify gaps for feature engineering. Able to develop/reuse data models & perform preprocessing, perform data mapping & implement data pipelines
• Technical Leadership: Provide technical guidance and mentorship to junior developers. Develop and maintain ETL routines using ETL and orchestration tools such as Airflow.
• Collaboration: Work closely with Architect, Platform owner, cross-functional team such as Business Analysts, product managers, and Data Scientists, to deliver high-quality data pipelines. Collaborate with data scientists to understand data sources & then integrate, design & implement scalable and efficient data pipelines.
• Documentation: Create and maintain data models, data mapping, technical documentation, including design documents and code documentation.

Qualifications & Skills:

• Bachelor's or Master's degree in Computer Science, Data Science, or a related field with Proven experience of 5+ years working as a Data Engineer, in a cloud based environment
• Proficiency of developing batch & streaming ETL pipelines using Python, SQL, Pyspark , open source format files (e.g., Iceberg/Hudi/Delta, Parquet, Avro, ORC) and spark optimisation
• Proven track record in leveraging or developing base frameworks, reusable programs and defining standards eg. Kedro, Great Expectations, etc.
• Experience of analysing, cleaning & transforming vast amounts of raw data from various systems using Spark to provide ready-to-use data to our data scientists, business analysts.
• Experience with cloud platforms AWS specifically S3, Lambda, EC2, EMR, RDS, AWS Glue, etc.
• Understanding of Distributed systems like Hadoop, Streaming - Kafka, Flink, Spark streaming
• Experience of designing & developing data architecture as per data lake paradigms
• Familiarity with Snowflake or other data warehousing platforms
• Experience with data mapping, data lineage, data modelling & understanding of data governance, data security practices
• Proficiency in database design and SQL and NoSQL databases like Redis, DynamoDB.
• Good understanding of the Agile methodologies and working in Scrum/SAFE practices
• Experience on Advanced Analytics project & understanding of AI/ML techniques
• Knowledge of software development best practices, including version control (Git) and continuous integration (CI/CD) processes.
• Strong problem-solving and debugging skills.
• Effective communication skills and the ability to work collaboratively with cross-functional teams.