Technical Lead - Data Engineering (10-14 yrs)

Role Overview :

We are looking for a Technical Lead to drive a large-scale Teradata-to-AWS Databricks migration and play a key role in designing and optimizing cloud-based data pipelines. The ideal candidate will have deep expertise in Databricks, ETL/ELT development, and relational databases, along with a strong background in performance tuning, automation, and team leadership.

Key Responsibilities :

- Lead and execute a large-scale Teradata-to-AWS Databricks migration strategy, ensuring a seamless and efficient transition.

- Architect, design, and implement scalable data pipelines leveraging Databricks tech stack, including Delta Lake, Spark, Unity Catalog, Databricks Workflows, and Lakehouse architecture.

- Develop and optimize ETL/ELT workflows, leveraging IBM DataStage, Apache Airflow, FiveTran, and other modern tools to streamline data processing.

- Strong expertise in Stored Procedures, PL/SQL, and SQL across relational databases like Oracle, SQL Server, and DB2, with experience in performance tuning and query optimization.

- Automate data ingestion, validation, and quality checks, ensuring high accuracy and efficiency in data processing.

- Drive best practices in data governance, security, and compliance, ensuring that cloud migration adheres to industry standards.

- Collaborate with senior architects, engineers, and business stakeholders to align the migration strategy with business objectives and technical feasibility.

- Lead and mentor a technical team, fostering an environment of innovation, continuous learning, and best practices in cloud data engineering.

- Work on performance tuning, security enhancements, and cost optimization to ensure efficient use of cloud resources.

- Stay up to date with emerging technologies in cloud data engineering, Databricks, and AI-driven optimizations to enhance workflows and improve efficiency