Data Engineer - Azure Databricks (4-7 yrs)

About the Role :

We are seeking a Data Engineer with expertise in Data Bricks and Azure Environments tools to join our dynamic team in Noida. In this role, you will design and implement scalable data pipelines, visualize insights, troubleshoot complex data-related challenges, and support the operational and analytical needs of our healthcare-focused solutions. As part of a collaborative team, you will play a critical role in ensuring data integrity, optimizing performance, and enabling data-driven decision-making.

Primary Responsibilities and Assignments :

Data Bricks Development and Administration :

- Design and implement end-to-end pipelines for data ingestion and processing.

- Develop schemas for Bronze, Silver, and Gold data layers to support robust data management.

- Manage user rights, permissions, and notebook maintenance.

- Troubleshoot advanced issues, including pipeline errors, capacity constraints, and schema evolution.

Cloud Integration :

- Work with Azure cloud-native tools, including Azure Data Factory, for seamless data ingestion and transformation.

Data Warehousing and Analytics :

- Define and maintain star schemas and apply Big Data concepts for optimized performance.

- Support analytical and reporting requirements with robust data models.

Database Management :

- Perform complex query optimization and relational database administration.

- Experience with MySQL is a plus.

Operational Support :

- Provide production-level support for pipelines and address critical issues outside of normal working hours when necessary.

Collaboration and Documentation :

- Work closely with cross-functional teams to ensure data solutions align with business objectives.

- Maintain comprehensive documentation for implemented solutions and processes.

- Secondary Responsibilities and Assignments

Data Visualization and Analytics :

- Create insightful dashboards and visualizations using Power BI to support business objectives.

- Integrate visualizations into business workflows to enhance decision-making.

Required Skills and Professional Background :

Technical Expertise :

- Proficiency with Data Bricks development and administration.

- Experience with data lookup tools for querying and analysing large datasets.

- Experience in designing schemas for data pipelines and warehousing layers.

- Advanced troubleshooting and optimization skills for pipelines and workflows.

Cloud and Data Warehousing :

- Hands-on experience with Azure cloud tools, particularly Azure Data Factory.

- Strong understanding of data warehousing principles and Big Data frameworks.

Database Knowledge :

- Advanced SQL skills with query formation and optimization.

- Experience with MySQL is an added advantage.

Education and Experience :

- Bachelor's degree in Computer Science, Data Science, Engineering, or a related field.

- Equivalent work experience may be considered in lieu of education requirements.

- Minimum 3 years of experience in data engineering or a similar field.