Azure Devops Engineer

Role & responsibilities

• Design, develop, and manage YAML-based CI/CD pipelines in Azure DevOps for the data migration lifecycle.
• Build, deploy, and monitor scalable data migration pipelines using Azure Data Factory, Azure Databricks, and Synapse Analytics.
• Implement Infrastructure as Code (IaC) using tools like Terraform, ARM Templates, and Bicep to provision and manage Azure resources.
• Collaborate with cross-functional teams to optimize pipelines for data validation, error handling, and monitoring during migration.
• Manage Azure resources such as Blob Storage, Data Lake, Virtual Networks, and Azure Functions for data staging and processing.
• Ensure data security through RBAC, Azure Key Vault, and compliance with organizational standards.
• Troubleshoot and optimize pipeline performance and resolve issues during the data migration process.
• Monitor and log data pipeline activities using Azure Monitor, Log Analytics, and Application Insights.

Preferred candidate profile

• 8+ years of experience in Azure DevOps with a focus in YAML-based CI/CD pipelines with multi-stage configurations.
• Integration of triggers (manual and automated) and approval gates.
• Leveraging reusable pipeline templates, conditional tasks, and environment variables.
• ETL/ELT Workflows: Knowledge of Azure Data Factory (ADF) and Azure Databricks for data migration and transformation.
• Data Validation: Scripts for data integrity checks during migration (Python, PowerShell, Bash).
• Experience with SQL Server, Cosmos DB, Azure Synapse Analytics, and Blob Storage for data handling and migration.
• Experience with Terraform and ARM Templates provisioning cloud resources.
• Azure Data Factory (ADF) for orchestrating data workflows.
• Azure Databricks for large-scale data transformations.
• Azure Synapse Analytics for data warehouse migration.
• Tools like SSMS, Azure Data Studio, and PostgreSQL Workbench for managing data migration tasks.
• Azure Monitor, Log Analytics, and Application Insights for pipeline and data flow monitoring.
• Azure Blob Storage and Data Lake for data staging and storage during migration.
• Setting up secure connections using Virtual Networks (VNETs) and Private Endpoints.
• Leveraging Azure VPN Gateway for hybrid cloud migrations.
• Azure Active Directory for authentication and RBAC for granular control.