MLOps Engineer - Kubernetes/Spark

Skill Set : Seldon Core, Cubeflow, CI/CD, Kubernetes, Prometheus, Grafana, ISTIO

Looking for an experienced and high-energy ML Ops Engineer. The primary function of this role is to design enterprise architecture. Envision and drive solution architecture after hearing the product s vision and user stories with ability to envision and drive a proactive architectural roadmap for an existing product keeping in mind the future requirements

Requirements :

- Holds experience in migration on spark jobs to Kubernetes.

- Excellent understanding of Open Source software and the Open Source Software community and the ability to collaborate

- Experience with containerizing and deploying ML models

- Sound knowledge and experience with at least one DL frameworks such as PyTorch, TensorFlow, Keras

- Experience using Python,TensorFlow, PyTorch and other Computer Vision frameworks

- Excellent understanding and experience with Kubernetes in deploying Cloud Native configured Applications, AWS EKS, Dockers and MLOps and DevOps pipeline

- Strong in providing solutions in data engineering and cloud migration projects, experienced as Architect for data governance, integration, modeling and quality management.

- Experience in enterprise data warehouse setup, Hadoop cluster installation, pre-sales and solution engineering recommendations.

Responsibilities :

- Seldon Core Setup and Installation in Kubernetes

- Seldon Core Usage for Alibi

- Seldon Core Ingress Setup with Istio

- Custom Kuberenetes version 1.17 (MicroK8S) installation procedure for supporting the deployment of Seldon Core

- Installation and access of OpenDistro ElasticSearch/Kibana with Kibana Dashboard Access - Enterprise Version including Multi-Tenancy and Security

- Integration and Networking of Kafka Cluster for Remote Access for the DeepStream Server.

- Reading Custom Indexes from ElasticSearch Dashboard.

- Setup of Monitoring Dashboards for Grafana/Prometheus for CV layer.

- Setup of Monitoring Dashboards for Grafana/Prometheus for Data layer.

- Setup of Monitoring of Infrastructure and logs using Logstash from ElasticSearch Dashboard.

- Pyspark Code integration into adding ETL data from Kafka topic into Cassandra tables

- Jenkins-X CI-CD installation and setup for Kubernetes Cluster for running CI-CD pipelines.

- Data layer setup using latest versions for Kafka, Apache Spark, Cassandra including any pre-packaged dashboards built-in.

- Istio Monitoring and Istio Dashboards for Service Mesh Topology.

- Dockerize and Kubernetes versions of React (Frontend) and NodeJS (Microservice) applications for Microservices architecture.

- Installation of Apache Pinot for Fast Indexing of Data in the Kafka Data Layer along with Dashboard access.

- Data Management and Data Lineage for Data layer and or CV layer using Pachyderm or Apache Atlas

- WSO2 API Manager Install/Setup configuration in Kubernetes