Senior Data Engineer

This role requires a well-rounded data engineer with hands-on experience in data processing technologies, a good understanding of data modeling concepts, and the ability to collaborate effectively with various stakeholders. The willingness to adapt to flexible working hours demonstrates a commitment to supporting the continuous operation of data pipelines and meeting business needs.

Responsibilities:

Build Data Pipelines:

• Utilize PySpark and Python to construct efficient and scalable data pipelines.
  
• Integrate data from multiple source systems into a unified target system.
  

Orchestrate Pipelines with Airflow:

• Use Apache Airflow to orchestrate and schedule data pipelines, ensuring timely and reliable execution.
  

Enhance Existing Pipelines:

• Understand existing data pipelines and make enhancements based on evolving business requirements.
  
• Implement improvements to optimize performance and maintainability.
  

Debugging and Root Cause Analysis:

• Troubleshoot and resolve any failures in data pipelines promptly.
  
• Conduct root cause analysis for pipeline failures and implement corrective measures.
  

Collaboration with Stakeholders:

• Work closely with various stakeholders, both within and across teams.
  
• Communicate effectively to understand and address business needs related to data processing.
  
• Weekend and Shift Support:
  
• Be available to work on weekends and in shifts if necessary to provide support for business operations.

Requirements

Experience:

• 3-5 years of experience as a data engineer, demonstrating a solid understanding of data engineering principles.

Technical Skills:

• Proficient in SQL, Python, and PySpark for designing and implementing data solutions.
  

• Knowledge of data warehousing techniques and dimensional modeling.

Orchestration Tools:

• Experience with Apache Airflow for orchestrating complex data workflows.
  

• Familiarity with containerization using Docker and version control systems.

Data Modeling and Transformation:

• Exhibit strong proficiency in data modeling techniques, emphasizing expertise in designing and implementing effective data structures.
  

• Knowledge of DBT (Data Build Tool) for transforming and modeling data.

Cloud Platform:

• AWS knowledge is a plus, showcasing familiarity with cloud-based data services and infrastructure