Data Engineer - Web Scraping (2-5 yrs)

Job Title : Data Engineer (Web Scraping) - Contract (2 Weeks, Extendable)

Location : New Delhi / Remote

Employment Type : Contract (2 Weeks, Extendable)

Experience : 2+ Years

Joining : Immediate

Job Overview :

We are looking for a Data Engineer with strong experience in web scraping to join our team as soon as possible. This is a short-term contract (2 weeks) with the possibility of an extension. The ideal candidate should have expertise in designing, building, and optimizing web scraping pipelines while handling large-scale data extraction efficiently.

Key Responsibilities :

- Develop and maintain scalable web scraping solutions using Python and related frameworks.

- Scrape data from various websites while handling challenges like CAPTCHA, IP blocking, and dynamic content.

- Optimize data extraction processes for efficiency, accuracy, and speed.

- Work with databases (SQL/NoSQL) to store and manage extracted data.

- Ensure data integrity, quality, and security throughout the extraction process.

- Automate scraping pipelines and schedule jobs for continuous data collection.

- Debug and resolve scraping failures, adapting to website structure changes.

Required Skills & Experience :

- 2+ years of experience in data engineering and web scraping.

- Strong proficiency in Python with experience in libraries like Scrapy, BeautifulSoup, Selenium, and Requests.

- Hands-on experience with handling anti-scraping mechanisms (CAPTCHA solving, rotating proxies, user-agent spoofing).

- Experience working with databases (SQL/NoSQL) for data storage and retrieval.

- Familiarity with cloud platforms like AWS, GCP, or Azure for data processing.

- Knowledge of ETL pipelines and data workflow automation.

- Ability to join immediately and work in a fast-paced environment.

Nice-to-Have :

- Experience with distributed scraping frameworks for large-scale data extraction.

- Knowledge of Docker, and Kubernetes for deployment.

- Familiarity with API-based data extraction and automation tools.

Why Join Us ?

- Short-term, high-impact role with potential for extension.

- Work on challenging web scraping projects with cutting-edge tools.

- Flexible location - New Delhi or Remote.

- Competitive compensation for the contract duration.