Glorious Insight - Senior Data Engineer - Azure Data Factory (3-6 yrs)

About the Role :

We are seeking a highly skilled and experienced Senior Data Engineer to join our growing team.

This is a full-time, onsite position located in Jewar (Noida).

As a Senior Data Engineer, you will play a crucial role in designing, building, and maintaining our data infrastructure.

You will be responsible for developing and optimizing ETL pipelines, data warehousing solutions, and data analytics capabilities.

This role requires a strong understanding of data engineering principles, experience with Azure cloud services, and proficiency in Python programming.

Responsibilities :

- Design, develop, and maintain robust and scalable data pipelines using Azure Data Factory and other relevant technologies.

- Implement ETL processes to extract data from various sources, transform it into usable formats, and load it into our data warehouse.

- Build and optimize data models for efficient storage and retrieval of information.

- Develop and deploy APIs for data ingestion and consumption, including experience with XML data processing through Function Apps and Event Hub.

- Utilize Azure Event Hub and Data Lake Containers for real-time data ingestion and storage.

- Implement DevOps CI/CD pipelines for automated deployment and management of data infrastructure.

- Collaborate with data scientists and analysts to understand their data needs and provide them with the necessary data resources.

- Monitor and troubleshoot data pipelines to ensure data quality and performance.

- Stay up-to-date with the latest data engineering trends and technologies.

- Contribute to the development of data governance and security policies.

- Mentor and guide junior data engineers.

- Proactively identify and address performance bottlenecks and data quality issues.

Required Skills and Experience :

- Extensive experience with Azure cloud services, including Azure Data Factory, Azure Event Hub, Azure Data Lake Containers, and Azure Functions.

- Proven ability to create and work with APIs, including experience making APIs and consuming XML data through APIs using Function Apps and Event Hub.

- Strong proficiency in Python, including Object-Oriented Programming principles and experience with Pydantic models.

- Deep understanding and hands-on experience in ETL processes and tools.

- Solid understanding of data warehousing concepts and experience building and maintaining data warehouses.

- Ability to design and implement efficient data models.

- Familiarity with DevOps practices and CI/CD pipelines.

- Excellent problem-solving and analytical skills.

Preferred Skills (Optional) :

- Experience with other cloud platforms (AWS, GCP).

- Knowledge of big data technologies (Spark, Hadoop).

- Experience with data visualization tools (Power BI, Tableau).

- Familiarity with data governance and security best practices