Data Engineer - SQL/Python (2-4 yrs)

About Freight Tiger :

Freight Tiger is India's largest software-enabled neutral freight network today. We have built anintelligent operating system, on-ground expertise, and a digital platform that together enhance every step of the logistics process. With our help, customers in traditional industries move material to their customers with full visibility, predictability and speed, all at lower cost. We do this through three products:

1. Cloud-based TMS (Transportation Management Software) & Visibility platform.

2. Freight Assignment platform to facilitate efficient freight procurement by Shippers via Logistics Service Providers.

3. Marketplace platform designed to seamlessly align demand and supply of trucks pan-india.

We take pride in processing an impressive 4% of India's total freight on our platform, serving a vast network of 400+ shippers, 1500+ logistics service providers (LSPs), across 90,000+ locations. Our exceptional clientele includes over 300 industry titans, among them Saint Gobain, Dalmia Group, JSW Steel, ITC, DHL, MRF, and Pernod Ricard. We have tier-1 investors such as Lightspeed Venture Partners, Munjal Family Trust, Florintree Infra (founded by Former CEO of Blackstone Group India), and Aroa Ventures (Family office of Oyo founder). Moreover, we have notable individual investors, such as Founders of Transporeon - Peter Forster & Marc-Oliver Simon, CEO of Transplace Francis McGuigan, Hermann Ude - Chairman of Transporeon and former CEO of DHL Global Freight Forwarding.

Company Size : Startup / Small Enterprise

Working Days : 5 days/week

About the Role :

We are seeking a highly skilled Senior Data Engineer with 2-7 years of experience to join our dynamic team. The ideal candidate will have a strong background in data engineering, with expertise in data warehouse architecture, data modeling, ETL processes, and building both batch and streaming pipelines. The candidate should also possess advanced proficiency in Spark, Databricks, Kafka, Python, SQL, and Change Data Capture (CDC) methodologies.

Key responsibilities :

- Design, develop, and maintain robust data warehouse solutions to support the organization's analytical and reporting needs.

- Implement efficient data modeling techniques to optimize performance and scalability of data systems.

- Build and manage data lakehouse infrastructure, ensuring reliability, availability, and security of data assets.

- Develop and maintain ETL pipelines to ingest, transform, and load data from various sources into the data warehouse and data lakehouse.

- Utilize Spark and Databricks to process large-scale datasets efficiently and in real-time.

- Implement Kafka for building real-time streaming pipelines and ensure data consistency and reliability.

- Design and develop batch pipelines for scheduled data processing tasks.

- Collaborate with cross-functional teams to gather requirements, understand data needs, and

deliver effective data solutions.

- Perform data analysis and troubleshooting to identify and resolve data quality issues and performance bottlenecks.

- Stay updated with the latest technologies and industry trends in data engineering and contribute to continuous improvement initiatives.

Ideal Candidate :

- Bachelor's degree in Computer Science, Engineering, or a related field.

- 5-6 years of hands-on experience in data engineering roles, with a focus on building and maintaining data warehouse solutions.

- Strong proficiency in data modeling concepts and experience with relational and dimensional data models.

- Extensive experience with ETL processes and tools for data ingestion, transformation, and loading.

- In-depth knowledge of Spark and Databricks for large-scale data processing and analytics.

- Proficiency in Kafka for building real-time streaming pipelines and event-driven architectures.

- Solid programming skills in Python for data manipulation, scripting, and automation tasks.

- Expertise in SQL for querying and analyzing data from relational databases and data warehouses.

- Experience with Change Data Capture (CDC) techniques and tools for capturing and propagating data changes.

- Excellent problem-solving skills and the ability to troubleshoot complex data engineering issues.

- Strong communication and collaboration skills, with the ability to work effectively in a team environment.