Data Engineer

• Hands-on experience in Azure Cloud services, Networking Concepts, Security - Cloud and on-premises system, Deployment using Azure DevOps, Azure Cloud Monitoring and Cost controls and Terraform - hands-on experience
• CI/CD Pipeline Management: Design, implement, and manage Continuous Integration/Continuous Deployment (CI/CD) pipelines using Azure DevOps and GitHub. Ensure the pipelines are efficient, reliable, and scalable.
• Infrastructure as Code (IaC): Automate the provisioning and management of infrastructure, with a focus on Azure Databricks and Azure Data Factory in a private network environment, using tools like Terraform and ARM templates.
• Environment Management: Create and manage development, testing, and production environments, ensuring consistency, security, and alignment with organizational requirements.
• Security: Implement security best practices throughout the CI/CD pipeline, including secrets management, secure code scanning, and compliance with security standards.
• Monitoring & Logging: Set up and maintain monitoring and logging for applications and infrastructure using Azure Monitor, Log Analytics, and related tools to ensure system reliability and performance.
• Automation: Identify opportunities for automation to streamline processes, reduce manual errors, and improve operational efficiency.
• Policy Enforcement: Establish and enforce policies such as branch policies, pull request reviews, and pipeline approvals to maintain code quality and compliance with organizational standards.
• Manage and maintain Azure Databricks Platform, workspaces, clusters, and jobs
• Oversee user administration including access controls and permissions
• Handle library installations, runtime management, and policy enforcement
• Implement/analyze cost control measures
• Administer Unity Catalog for data governance and security
• Collaborate with data engineers, data scientists, and analysts to optimize and streamline data workflows, and analytical pipelines on the Databricks platform
• Manage Azure storage solutions, including Blob Storage and Data Lake Storage
• Administer Azure Key Vault for secure storage of secrets and keys
• Configure and manage Azure Data Factory for data integration and ETL processes
• Implement and manage VNETs, firewalls, Azure policies and security best practices
• Set up budgets and alerts to monitor and control Azure costs and spend
• Configure alerts for proactive issue detection and resolution
• Databricks Lakehouse Monitoring

Minimum Qualifications

• Minimum of Bachelor's degree in Computer science, Information systems, or Engineering
• Experience: 7+ years of professional experience in DevOps, with a strong focus on Azure. Knowledge of Data Platforms like Databricks, Data Factory (preferred)
• Proficiency in scripting languages such as Python/PySpark, PowerShell, or Bash
• Experience in automating administrative tasks and workflows
• Knowledge of security best practices and compliance requirements
• Experience with ETL processes, data pipelines, and big data technologies
• Experience with backup and restore procedures for Databricks and Azure services
• Ability to troubleshoot and resolve issues in a timely and efficient manner
• Strong verbal and written communication skills
• Ability to document processes, procedures, and configurations clearly
• Commitment to providing high-quality support to internal/external users and stakeholders
• Ability to understand and address the needs of internal and external customers
• Team player with the ability to work collaboratively with cross-functional teams
• Flexibility to adapt to changing requirements and priorities

Offshore data engineer with Databricks ML (leverage Python skill to train as needed). 

• Continuous Learning: Eagerness to learn new technologies and continuously improve skills to stay current in a rapidly evolving field.

Preferred Qualifications

Offshore data engineer with Databricks ML (leverage Python skill to train as needed). 

• Deep understanding of Databricks architecture, features, and best practices
• Experience with Delta Lake and Databricks SQL for advanced data management
• Experience in performance tuning and optimization for both Databricks and Azure environments
• Experience supporting/administering data science and ML workloads in Databricks
• Knowledge of disaster recovery planning and implementation
• Experience integrating Databricks with other data platforms and tools
• Understanding of hybrid and multi-cloud environments
• Experience implementing data protection measures and ensuring regulatory compliance