Jr Data Engineer - Web Scraping/ Web Crawling

Job Title:- Junior Data Engineer Web Scraping/Web Crawling
Experience :-1 to 4 Years

Job Overview: We are looking for a highly motivated Junior Data Engineer with a passion for
web scraping and web crawling to join our team. The ideal candidate will have strong Python
programming skills and experience with web scraping frameworks and libraries like Requests,
BeautifulSoup, Selenium, Playwright or URLlib. You will be responsible for building
efficient and scalable web scrapers, extracting valuable data, and ensuring data integrity.

This role requires a keen eye for problem-solving, the ability to work with complex data
structures, and a strong understanding of web technologies like HTML, CSS, DOM,
XPATH, and Regular Expressions. Knowledge of JavaScript would be an added
advantage.

• Responsibilities:
• As a Web Scraper, your role is to apply your knowledge set to fetch data from multiple online sources
• Developing highly reliable web Scraper and parsers across various websites
• Extract structured/unstructured data and store them into SQL/No SQL data store
• Work closely with Project/Business/Research teams to provide scrapped data for analysis
• Maintain the scraping projects delivered to production
• Develop frameworks for automating and maintaining constant flow of data from multiple sources
• Work independently with minimum supervision
• Develop a deep understanding of the data sources on the web and know exactly how, when, and which data to scrap, parse and store this data
  
• Required Skills and Experience:
  Experience as Web Scraper of 1 to 4 years.
• Proficient knowledge in Python language and working knowledge of Web Crawling/Web scraping in Python Requests, Beautifulsoup or URLlib and Selenium, Playwright.
• Must possess strong knowledge of basic Linux commands for system navigation, management, and troubleshooting.
• Must have expertise in proxy usage to ensure secure and efficient network operations.
• Must have experience with captcha-solving techniques for seamless automation and data extraction.
• Experience with data parsing - Strong knowledge of Regular expression, HTML, CSS, DOM, XPATH. Knowledge of Javascript would be a plus
• Must be able to access, manipulate, and transform data from a variety of database and flat file sources. MongoDB & MYSQL skills are essential.
• Must possess strong knowledge of basic Linux commands for system navigation,
  management, and troubleshooting.
• Must be able to develop reusable code-based scraping products which can be used by others.
• GIT knowledge is mandatory for version control and collaborative development workflows.
• Must have experience handling cloud servers on platforms like AWS, GCP, and LEAPSWITCH for scalable and reliable infrastructure management.
• Ability to ask the right questions and deliver the right results in a way that is understandable and usable to your clients.
• A track record of digging in to the tough problems, attacking them from different angles, and bringing innovative approaches to bear is highly desirable. Must be capable of self- teaching new techniques.
  

Behavioural expectations:

• Be excited by and have positive outlook to navigate ambiguity.
• Passion for results and excellence
• Team player
• Must be able to get the job done by working collaboratively with others
• Be inquisitive and an analytical mind; out-of-the-box thinking
• Prioritize among competing opportunities, balance consumer needs with business and product priorities, and clearly articulate the rationale behind product decisions
• Straightforward and professional
• Good communicator
• Maintain high energy and motivate
• A do-it-yourself orientation, consistent with the company and "roll-up the- sleeves" culture.