AI/ML Engineer - Python/PyTorch (6-7 yrs)

Job Title - AI/ML Engineer

Experience - 6+ years

Location - Hyderabad, Vizag

Primary Skills - Python, PyTorch, TensorFlow, LLM Models

Roles and Responsibilities :

1. Data Collection, Preprocessing, and Management :

Data Collection : Gather diverse datasets for training LLMs or other machine learning models. This may include text data from various sources like web scraping, databases, or APIs.

Data Preprocessing : Clean and preprocess raw text data (e.g., tokenization, stemming, lemmatization, removing stopwords) for NLP tasks using tools such as SpaCy, NLTK, or custom preprocessing pipelines.

Data Augmentation : Create synthetic data or perform augmentation techniques to enrich training datasets, particularly in scenarios where large labeled datasets are scarce.

Data Pipeline Development : Build automated and scalable data pipelines for continuous data ingestion, cleaning, and feeding into models.

2. Building and Training Machine Learning Models :

Model Selection and Design : Design and implement deep learning architectures for specific use cases (e.g., LLMs for NLP tasks like sentiment analysis, text summarization, question answering).

Model Development Using PyTorch and TensorFlow : PyTorch : Build and train custom neural networks using PyTorch, leveraging its dynamic computation graph and flexibility for research and experimentation.

TensorFlow : Implement scalable, production-ready models using TensorFlow (including TensorFlow Hub and Keras for high-level model building).

Training Large Models : Train large models like transformers (e.g., BERT, GPT, T5) using large-scale datasets. Efficiently handle high computational requirements for these models, potentially using cloud services (AWS, GCP) or GPUs.

Fine-Tuning Pre-trained Models : Fine-tune pre-trained models like BERT, GPT-3, or other LLMs on task-specific data to improve performance on downstream applications.

Model Evaluation : Use evaluation metrics like accuracy, F1 score, BLEU score (for text generation), or perplexity to assess model performance. Perform cross-validation and hyperparameter optimization.

3. Model Optimization and Scaling :

Hyperparameter Tuning : Experiment with hyperparameters (e.g., learning rates, batch sizes, number of layers, dropout rates) to enhance model performance and prevent overfitting.

Optimization : Use model optimization techniques such as quantization, pruning, and knowledge distillation to reduce the size and improve the inference speed of large models.

Distributed Training : Implement distributed training using PyTorch Distributed or TensorFlow's MirroredStrategy to train large models efficiently across multiple GPUs/TPUs.

4. Model Deployment and Integration :

Model Deployment : Deploy AI/ML models into production environments (e.g., AWS SageMaker, Google AI Platform) ensuring scalability, security, and robustness.

API Development : Build APIs or microservices for serving models, enabling real-time predictions or batch processing using frameworks like Flask, FastAPI, or TensorFlow Serving.

Model Monitoring : Implement monitoring systems to track the performance and accuracy of models in production. Detect model drift or degradation over time and retrain when necessary.

Scalability and Optimization : Ensure that the models can scale to handle large-scale inference workloads. Use TensorFlow Lite for edge devices or ONNX for cross-framework deployment.

5. Research and Experimentation :

Cutting-Edge Research : Stay up to date with the latest advancements in machine learning, especially in transformer models and NLP, and incorporate state-of-the-art techniques into your work.

Innovation : Experiment with novel approaches for improving model accuracy, efficiency, or generalization (e.g., new transformer variants, unsupervised pretraining techniques).

Contributing to Open Source : Contribute to or develop open-source projects that enhance machine learning tools, especially in the field of NLP and LLMs.