DevOps/MLOps Engineer - Hadoop/Spark (8-10 yrs)

Job Purpose :

We are seeking a highly skilled and experienced DevOps/MLOps Engineer to join our dynamic team. The ideal candidate will have 8+ years of IT experience with a strong focus on DevOps and MLOps. You will be responsible for building, automating, and maintaining robust, scalable, and secure infrastructure to support both software development and machine learning workflows.

Collaborating closely with development, data science, and operations teams, you will ensure the seamless integration, deployment, and monitoring of software and ML models in production environments.

Roles and Responsibilities :

Infrastructure Management :

- Oversee the end-to-end deployment pipeline, from server provisioning to database and networking management.

- Automate infrastructure provisioning, configuration management, and operational tasks using tools such as Terraform, Ansible, or scripting.

- Implement system monitoring, log management, and troubleshooting processes to ensure optimal performance and stability.

- Enforce security best practices, including secure coding, encryption, and vulnerability scanning.

DevOps Engineering :

- Design, implement, and manage CI/CD pipelines to streamline software releases with automated testing and deployment.

- Manage and maintain cloud infrastructure (AWS, Azure, or GCP) focusing on cost optimization, security, and high availability.

- Implement Infrastructure as Code (IaC) using tools like Terraform or CloudFormation.

- Automate infrastructure provisioning, configuration management, and application deployments.

- Implement and maintain monitoring and logging systems (e.g., Prometheus, Grafana, ELK stack).

- Collaborate with development teams to enhance software development and release processes.

- Promote DevOps best practices and foster a culture of automation and continuous improvement.

MLOps Engineering :

- Design and implement MLOps pipelines for model training, deployment, and monitoring.

- Automate model training and hyperparameter tuning processes.

- Deploy and manage machine learning models in production environments.

- Monitor model performance and data drift, implementing retraining pipelines as needed.

- Build and maintain scalable and reliable ML model serving infrastructure.

- Integrate ML models into existing applications and workflows.

- Collaborate with data scientists to translate their requirements into efficient and scalable infrastructure.

- Implement model versioning, experiment tracking, and data security throughout the ML lifecycle.

General Responsibilities :

- Participate in on-call rotations to provide system support.

- Contribute to technical documentation and knowledge-sharing initiatives.

- Stay up to date with the latest DevOps and MLOps trends and technologies.

- Mentor junior engineers and foster a culture of learning and improvement.

Required Skills :

Cloud Platforms : Expertise in at least one major cloud provider (AWS, Azure, or GCP); multi-cloud experience is a plus.

Containerization & Orchestration : Proficiency with Docker and Kubernetes.

Infrastructure as Code (IaC) : Experience with Terraform or CloudFormation.

CI/CD : Expertise in Jenkins, GitLab CI, CircleCI, or GitHub Actions.

Configuration Management : Experience with Ansible, Chef, or Puppet.

Monitoring & Logging : Familiarity with Prometheus, Grafana, CloudWatch, Azure Monitor, ELK stack, or Splunk.

Scripting : Strong proficiency in Python, Bash, or other scripting languages.

Operating Systems : Deep understanding of Linux-based systems.

MLOps Tools : Experience with MLflow, Kubeflow, SageMaker, or Azure ML.

Machine Learning : Basic understanding of ML concepts and algorithms.

Version Control : Proficiency in Git.

Communication & Collaboration : Excellent communication skills with a focus on cross-functional teamwork.

Preferred Skills :

- Experience with serverless technologies (AWS Lambda, Azure Functions).

- Knowledge of data engineering tools (Spark, Hadoop, Kafka, etc.).

- Experience working with SQL and NoSQL databases.

- Expertise in security best practices and implementations.

- Experience in an Agile development environment.

Required Qualifications :

- Bachelor's or Master's degree in Computer Science, Information Technology, or a related field.

- 8+ years of IT experience, with a strong focus on DevOps and MLOps.