Senior Data Engineer

About the Role

We are seeking a Senior Data Engineer to design, develop, and optimize large-scale data pipelines, transformation workflows, and analytics-ready datasets. This role requires expertise in Python, Apache Airflow, Apache Spark, SQL, and DuckDB, along with strong experience in data quality, data processing, and automation.

As a Senior Data Engineer, you will play a key role in building scalable, high-performance data engineering solutions, ensuring data integrity, and supporting real-time and batch data workflows. You will work closely with Data Scientists, Analysts, DevOps, and Engineering teams to build efficient, cost-effective, and reliable data architectures.

Key Responsibilities

• Design, build, and maintain scalable ETL/ELT data pipelines using Apache Airflow, Spark, and SQL.
• Develop Python-based data engineering solutions to automate data ingestion, transformation, and validation.
• Implement data transformation and quality checks for structured and unstructured datasets.
• Work with DuckDB and other in-memory databases to enable fast exploratory data analysis (EDA).
• Optimize data storage and retrieval using Parquet, Apache Iceberg, and S3-based data lakes.
• Develop SQL-based analytics workflows and optimize performance for querying large datasets.
• Implement data lineage, governance, and metadata management for enterprise-scale data solutions.
• Ensure high availability, fault tolerance, and security of data pipelines.
• Collaborate with Data Science, AI/ML, and Business Intelligence teams to enable real-time and batch analytics.
• Work with cloud platforms (AWS, Azure, GCP) for data pipeline deployment and scaling.
• Write clean, efficient, and maintainable code following best software engineering practices.

Required Skills & Qualifications

• 7+ years of experience in data engineering, big data processing, and backend development.
• Expertise in Python for data processing and automation.
• Strong experience with Apache Airflow for workflow orchestration.
• Hands-on experience with Apache Spark for big data transformations.
• Proficiency in SQL (PostgreSQL, DuckDB, Snowflake, etc.) for analytics and ETL workflows.
• Experience with data transformation, data validation, and quality assurance frameworks.
• Hands-on experience with DuckDB, Apache Arrow, or Vaex for in-memory data processing.
• Knowledge of data lake architectures (S3, Parquet, Iceberg) and cloud data storage.
• Familiarity with distributed computing, parallel processing, and optimized query execution.
• Experience working in CI/CD, DevOps, containerization (Docker, Kubernetes), and cloud environments.
• Strong problem-solving and debugging skills.
• Excellent written and verbal communication skills.

Preferred Skills (Nice to Have)

• Experience programming in JAVA/JEE platform is highly desired.
• Experience with data streaming technologies (Kafka, Flink, Kinesis).
• Familiarity with NoSQL databases (MongoDB, DynamoDB).
• Exposure to AI/ML data pipelines and feature engineering.
• Knowledge of data security, compliance (SOC2 Type2, GDPR, HIPAA), and governance best practices.
• Experience in building metadata-driven data pipelines for self-service analytics.