Python Developer (Web Scraping, LLMs, Cloud, AI/ML)

Role & responsibilities

• Develop web scraping solutions using Python, Selenium, and other scraping tools to collect data from various websites, managing structured (e.g., JSON, XML) and unstructured data (e.g., PDFs, text documents).
• Build and maintain ETL pipelines using AWS Glue, PySpark, and Pandas to clean, transform, and load scraped data for analytics, ensuring data integrity and high performance.
• Manage AWS Glue Catalog, Glue Jobs, and Workflows to orchestrate and automate cloud-based data processing for large-scale scraping operations.
• Implement Large Language Models (LLMs) such as GPT, BERT, T5, and RAG (Retrieval-Augmented Generation) for advanced data extraction, text summarization, and entity recognition from unstructured sources.
• Apply LLMs for NLP tasks such as text classification, document parsing, and content generation across different data formats like PDFs and text documents.
• Use OCR technologies and models for extracting text from scanned documents and PDFs.
• Design and maintain cloud-based workflows using AWS EC2, Lambda, S3, CloudWatch, and Athena to manage, optimize, and scale scraping operations efficiently.
• Work with semi-structured (e.g., XML, JSON) and structured data to build robust ingestion and processing workflows.
• Utilize cloud-native tools and compression strategies like Snappy and Gzip to optimize performance for large-scale datasets.
• Automate and enhance scraping workflows by integrating LLMs and AI/ML for smarter data analysis, classification, and reporting.

Preferred candidate profile

We are seeking an experienced Python Developer with a strong background in web scraping, Large Language Models (LLMs), and cloud-based ETL workflows to join our dynamic engineering team. This position will involve developing and maintaining sophisticated data extraction pipelines that process unstructured, semi-structured, and structured data. The ideal candidate will have a proven track record of utilizing AI/ML models, particularly LLMs, to streamline and optimize data workflows.

Preferred Qualifications:

• Advanced degree in Computer Science, Data Science, Engineering, or related field.
• Experience with Docker, Kubernetes, or other containerization technologies.
• Knowledge of distributed computing and parallel processing.
• Experience with NLP (Natural Language Processing) tasks and tools such as spaCy, Hugging Face Transformers, or similar.
• Familiarity with data pipelines and orchestration tools like Apache Airflow, Prefect, or similar.
• Contributions to open-source projects or personal GitHub repositories showcasing your expertise.