Azure Senior Data Engineer

Description

The Azure Data Engineer at SG Analytics will leverage strong data infrastructure and architecture skills to lead Azure-based data pipeline projects. This role involves collecting, managing, analysing, and visualizing large datasets, ensuring data quality, and optimizing solutions using Azure tools, Python, PySpark, SQL, and CI/CD practices. The engineer will collaborate with data scientists, business analysts, and leadership to drive cloud-based strategies, automate processes, and support mission-critical applications, while maintaining security and efficiency in cloud environments.

Responsibilities:

• Master the use of technology and modern tools to optimize solutions and continuous process improvements.
• Facilitate the design, build, and testing of end-to-end data pipelines, including data ingestion (streaming, events, and batch), data integration, and data curation.
• Implement job and testing automation.
• Support mission-critical applications and near real-time data needs from the data platform.
• Build and support reusable frameworks for data ingestion, integration, and provisioning.
• Capture and publish metadata and new data to subscribed users.
• Work collaboratively with product managers, data scientists, and business partners, actively participating in design thinking sessions.
• Motivate, coach, and serve as a role model and mentor for other development team members leveraging the platform.
• Establish relationships with key business lines, Infosec, and infrastructure teams.
• Ensure security is integrated into all cloud services.
• Build, monitor, and manage the cloud infrastructure.
• Monitor and optimize cloud workloads.
• Review current and future Azure implementations and suggest improvements and standardizations for cloud offerings.

Required Skills:

• Bachelors degree in computer science, Physics, Mathematics, Statistics, or other Engineering disciplines.
• 4+ years' experience in Azure.
• 2-3 years' experience with data pipeline and workflow management tools: Azure Data Factory, SSIS.
• Experience with object-oriented/object function scripting languages: Python, PySpark, SQL, etc.
• Experience with systems scripting languages: Batch Scripts, Powershell.
• Experience with Git/GitHub and automating builds.
• Strong interpersonal, verbal, and written communication skills, with the ability to present complex technical/analytical concepts to an executive audience.
• Strong business mindset with customer obsession; ability to collaborate with business partners to identify needs and opportunities for improved data management and delivery.

Preferred Skills:

• Experience performing root cause analysis on internal and external data and processes to answer specific business questions and identify opportunities for improvement.
• Experience with the Agile workflow, Test Driven Development, and continuous improvement/continuous deployment.
• Strong analytic skills related to working with structured and unstructured datasets.
• Build processes supporting data transformation, data structures, metadata, dependency, and workload management using Azure Automation, Azure Function Apps, and Azure Data Factory.
• A successful history of manipulating, processing, and extracting value from large, disconnected datasets.
• Working knowledge of Azure Automation, Azure Data Factory, and highly scalable 'big data' data stores such as Azure Blob Storage, Azure Data Lake.
• Experience supporting and working with cross-functional teams in a dynamic environment.