Opening Vector DB Engineer

Job Description:

A skilled Analytics Engineer / Data Analyst is sought with expertise in vector search algorithms. The role focuses on improving the performance and efficiency of algorithms that identify similarities between data points based on vector representations. The ideal candidate will have a strong background in data analysis, machine learning, and algorithm optimization, with a specific focus on working with vector-based models for search and similarity.

Key Responsibilities:

• Analyze and optimize the performance of the existing vector search algorithm to identify and rank similar data points or items based on vector similarity.
• Process large datasets using vector representations (e.g., word embeddings, sentence embeddings) to enable advanced search and recommendation capabilities.
• Work closely with data scientists, engineers, and product teams to integrate vector search models into the broader system and ensure seamless performance.
• Work on fine-tuning vector representations, similarity measures, and indexing mechanisms to improve search efficiency.
• Utilize advanced data analytics techniques to extract insights and make data-driven recommendations based on the results of vector search algorithms.
• Profile and benchmark the vector search process, identifying bottlenecks, proposing solutions and optimizing critical parts of the search pipeline.
• Communicate findings and insights from vector search data to stakeholders through clear data visualizations, reports, and presentations.
• Conduct experiments to test new approaches for vector search, comparing various models and strategies to identify the most effective methods.
• Continuously monitor performance and suggest improvements based on real-time data.

Essential Skills and Qualifications:

• Strong understanding of statistical analysis, similarity search, and distance metrics (e.g., cosine similarity, Euclidean distance).
• Proven experience in implementing vector search algorithms (e.g., k-NN, approximate nearest neighbors (ANN)) to identify similar items based on vector similarity.
• Familiarity with databases (both SQL and NoSQL) and big data tools for handling large-scale datasets, such as Pandas, NumPy, and Spark.
• Proficiency with vector search frameworks or distributed computing techniques.
• Experience in Python and AWS.

Nice to have:

• Hands-on experience with Elasticsearchs dense vector fields, KNN search capabilities, FAISS, Voyager, etc for fast and scalable similarity searches is a plus.
• Experience with high-performance computing, distributed systems, and large-scale data processing.
• Expertise in optimizing machine learning algorithms or recommendation systems.
• Experience in profiling and optimizing the performance of search algorithms for low latency and high throughput.
• Hands-on experience with machine learning techniques, especially related to recommendation systems or search algorithms.