Osfin - Data Platform Engineer (5-7 yrs)

Job Description :

Responsibilities :

- Design and implement scalable and reliable data platform architectures, including data lakes, data warehouses, and data pipelines.

- Evaluate and select appropriate technologies and tools for data ingestion, storage, processing, and analysis.

- Define and enforce data governance and security policies.

- Build and maintain robust and efficient data pipelines for batch and real-time data processing.

- Develop and optimize ETL/ELT processes using tools like Apache Spark, Airflow, Kafka, or similar technologies.

- Implement data quality checks and monitoring to ensure data accuracy and consistency.

- Manage and maintain cloud-based data infrastructure (e.g., AWS, Azure, GCP), including provisioning, monitoring, and optimization.

- Ensure high availability and disaster recovery for the data platform.

- Monitor and optimize the performance of data pipelines and infrastructure.

- Troubleshoot and resolve data-related issues, including performance bottlenecks and data quality problems.

- Conduct performance testing and capacity planning.

Qualifications :

Experience :

- 5+ years of experience in data platform engineering or related roles.

- Proven experience in designing and building scalable data platforms.

- Extensive experience with cloud-based data infrastructure (AWS, Azure, or GCP).

- Proficiency in SQL and at least one programming language (Python, Java, Scala).

- Strong understanding of data warehousing and data lake concepts.

- Experience with ETL/ELT tools and technologies (e.g., Apache Spark, Airflow, Kafka, Flink, Dataflow, Glue).

- Experience with cloud-based data services (e.g., Redshift, BigQuery, Snowflake, Databricks, Azure Synapse).

- Experience with data governance and security practices.

- Experience with containerization and orchestration tools (Docker, Kubernetes) is a plus.

- Experience with CI/CD pipelines is a plus.

Education : Bachelor's or Master's degree in Computer Science, Engineering, or a related field.

Preferred Qualifications :

- Contributions to open-source data projects.

- Relevant certifications (e.g., AWS Certified Data Analytics - Specialty, Google Cloud Professional Data Engineer).