Python Engineer - Data Engineering and LLM API Development

Role Overview:

As a Python Engineer, you will play a critical role in building and scaling data pipelines, developing prompts for large language models (LLMs), and deploying them as efficient, scalable APIs. You will collaborate closely with data scientists, product managers, and other engineers to ensure seamless integration of data solutions and LLM functionalities. This role requires expertise in Python, API design, data engineering tools, and a strong understanding of LLMs and their applications.

Key Responsibilities

1. Data Engineering

• Build and maintain scalable data preparation pipelines using Python and relevant tools.
• Optimize data workflows to handle large datasets efficiently.
• Work with structured, semi-structured, and unstructured data sources to preprocess and transform data for LLM consumption.

2. LLM Prompt Development

• Develop, test, and refine prompts to optimize LLM outputs for specific use cases.
• Collaborate with data scientists to fine-tune LLMs or utilize pre-trained models effectively.
• Research and implement best practices for prompt engineering to achieve high-quality results.

3. API Hosting

• Design and deploy RESTful APIs to host LLM-based solutions.
• Ensure APIs are secure, performant, and scalable to handle high traffic.
• Integrate APIs with monitoring and logging solutions to track performance and issues.

4. Collaboration and Documentation

• Collaborate with cross-functional teams to gather requirements and deliver solutions.
• Write comprehensive technical documentation, including API specs and data pipeline designs.

Required Skills

• Strong Python programming skills with experience in building production-grade applications.
• Experience with data engineering tools such as Pandas, PySpark, Airflow, or similar.
• Hands-on experience with LLMs (e.g., OpenAI, Hugging Face) and prompt development.
• Proficiency in designing and deploying APIs using frameworks like FastAPI or Flask REST Framework.
• Familiarity with cloud platforms (AWS, Azure, or GCP) and containerization tools like Docker.
• Knowledge of SQL and experience with relational databases like PostgreSQL, MySQL, etc.
• Basic understanding of API security (e.g., OAuth, JWT) and scalability strategies.

Nice to Have

• Experience with vector databases like Pinecone, Weaviate, or FAISS for LLM embeddings.
• Familiarity with MLOps tools like MLflow or Kubeflow.
• Exposure to streaming technologies such as Kafka or Spark Streaming.
• Knowledge of LLM fine-tuning techniques using frameworks like PyTorch or TensorFlow.
• Experience with monitoring tools such as Prometheus, Grafana, or Datadog.

Soft Skills

• Strong problem-solving abilities with a proactive mindset.
• Excellent communication skills to explain technical concepts to non-technical stakeholders.
• Ability to work collaboratively in a cross-functional team.
• Passion for exploring new technologies and continuously learning.