Senior Data Engineer

Role & responsibilities

Data Engineering & Pipeline Development

• Design, build, and optimize ETL/ELT pipelines using Databricks, PySpark, and SQL.
• Implement incremental data ingestion and transformation frameworks using Delta Lake and Medallion Architecture (Bronze, Silver, Gold layers).
• Develop high-performance batch and real-time data processing solutions for analytics and reporting.

2. Data Architecture & Performance Optimization

• Develop and maintain scalable and cost-efficient data architectures on cloud platforms (Azure, AWS).
• Optimize Spark-based processing for efficiency, scalability, and cost reduction.
• Implement data partitioning, indexing, and query tuning techniques for performance optimization.

3. Data Modeling & Governance

• Design and maintain conceptual, logical, and physical data models for structured and semi-structured data.
• Develop dimensional models (Star Schema, Snowflake Schema) for analytics and BI reporting.
• Ensure data integrity, quality, and security by implementing governance policies and best practices.

4. Collaboration & Communication

• Work closely with business analysts, data scientists, and stakeholders to translate business needs into technical solutions.
• Communicate complex data concepts effectively to both technical and non-technical audiences.
• Provide technical leadership and mentor junior data engineers.

Preferred candidate profile

• 4+ years of experience in data engineering, data architecture. 
• Extensive hands-on experience with Databricks, Delta Lake, and Apache Spark.
• Strong knowledge of SQL-based data processing, query optimization, and indexing.
• Proficiency in Python, PySpark, and data engineering workflows.
• Experience with cloud platforms (AWS, Azure) and data lake/lakehouse architectures.
• Hands-on experience with data governance, metadata management, and data security.
• Solid understanding of CI/CD for data pipelines and infrastructure-as-code (Terraform, CloudFormation, etc.).
• Strong logical thinking, problem-solving, and analytical abilities.
• Excellent communication and stakeholder management skills.