Data Engineer

Job Description

Exp Range: 5 to 10 yrs

Location : PAN India

Key Competencies:

1. Generative AI & LLM Expertise:

• Demonstrated experience with generative AI models, particularly LLMs for tasks like text analysis, similarity detection, summarization, and response generation.
• Experience in developing and fine-tuning LLMs for text analysis, response generation, and summarization.
• Skills in prompt engineering to extract insights and manage complex text interactions.
• Familiarity with handling multimodal data inputs (text, images, logs, attachments).

1. Data Engineering Skills:

• Strong capabilities in building data pipelines for data ingestion, preprocessing, and transformation.
• Proficiency in building ETL pipelines to process and ingest large datasets, including text and image data.
• Experience with data cleaning and preprocessing for diverse file types (logs, images, documents).
• Strong skills in data categorization, tagging, vectorization, and building data workflows for LLM-based solutions.

1. Tool and Framework Proficiency:

• Proficiency with Python and relevant AI/ML libraries.
• Familiarity with MS Azure AI services, including Azure AI Studio, Azure AI Search, and vector store integrations.
• Hands-on experience with LangChain, AutoGen, Semantic Kernel, and other frameworks for building scalable AI solutions.
• Knowledge of embedding techniques, vectorization, and vector store technologies (e.g., FAISS, Pinecone) for efficient similarity searches and indexing.

1. Data Integration and Workflow Automation

• Strong understanding of data workflows that integrate text, image, and log processing.
• Ability to build solutions that support real-time data updates and interactive status tracking.
• Experience developing tools for automated data annotation and tagging, contributing to improved data quality and LLM training processes.

1. Analytical and Dashboard Capabilities:

• Skills in developing reports and dashboards using LLM outputs.
• Proficiency in integrating insights into visual tools for monitoring project statuses and trends.
• Ability to create tools for real-time tracking and updates on bug statuses.

1. Problem-Solving and Business Communication:

• Ability to work closely with stakeholders to understand use cases and translate them into effective AI solutions.
• Strong problem-solving skills to manage complex workflows and ensure accurate categorization and summarization.
• Effective communication for collaborating on enhancement needs and improvements.