Lead - Data Engineer

Manage cloud data lake development and deployment on AWS /Databricks.

• Databricks Platform Admin: Manage and create workspaces, configure cloud resources, view usage data, and manage account identities, settings, and subscriptions in Databricks

Development responsibilities:

• Optimize data pipelines for performance and efficiency.
• Work with Databricks clusters and configuration management tools.
• Use appropriate tools in the cloud data lake development and deployment. Developing/implementing cloud infrastructure to support current and future business needs.
• Provide technical expertise and ownership in the diagnosis and resolution of issues.
• Ensure all cloud solutions exhibit a higher level of cost efficiency, performance, security, scalability, and reliability.

Technical Skills:

• Experience & Proficiency with Databricks platform - Delta Lake storage, Spark (PySpark, Spark SQL). Must be well versed with Databricks Lakehouse, Unity Catalog concept and its implementation in enterprise environments.
• Familiarity of data design pattern - medallion architecture to organize data in a Lakehouse.
• Experience & Proficiency with AWS Data Services - S3, Glue, Athena, Redshift etc.| Airflow scheduling
• Proficiency in SQL and experience with relational databases.
• Proficiency in at least one programming language (e.g., Python, Java) for data processing and scripting.
• Experience with DevOps practices - AWS DevOps for CI/CD, Terraform/CDK for infrastructure as code

Good understanding of data principles, Cloud Data Lake design & development including data ingestion, data modeling and data distribution.

Jira:

Proficient in using Jira for managing projects and tracking progress.

Other Skills:

• Strong communication and interpersonal skills.
• Collaborate with data stewards, data owners, and IT teams for effective implementation
• Understanding of business processes and terminology
• Experienced with Scrum and Agile Methodologies

Qualification

• Bachelor s degree in information technology or a related field. Equivalent experience may be considered.
• Overall experience of 8-12 years in Data Engineering