Lead Data Engineer

Role & responsibilities

• Developing Modern Data Warehouse solutions using Databricks and Azure Stack
• Ability to provide solutions that are forward-thinking in data engineering and analytics space
• Collaborate with DW/BI leads to understand new ETL pipeline development requirements.
• Triage issues to find gaps in existing pipelines and fix the issues
• Work with business to understand the need in reporting layer and develop data model to fulfill reporting needs
• Drive technical discussion with client architect and team members
• Orchestrate the data pipelines in scheduler via Airflow Skills and Qualifications:
• Bachelor's and/or masters degree in computer science or equivalent experience.
• Good understanding of Databricks Data & AI platform and Databricks Delta Lake Architecture
• Should have hands-on experience in SQL, Python and Spark (PySpark)
• Experience in building ETL / data warehouse transformation processes
• Proficiency in RDBMS, complex SQL, PL/SQL, Unix Shell Scripting, performance tuning and troubleshoot
• Databricks Certified Data Engineer Associate/Professional Certification (Desirable).
• Comfortable working in a dynamic, fast-paced, innovative environment with several ongoing concurrent projects
• Should have experience working in Agile methodology
• Strong verbal and written communication skills.
• Strong analytical and problem-solving skills with a high attention to detail.

Mandatory Skillset : Pyspark, Azure Databricks, SQL