Softpath Technologies - Senior Software Engineer - Web Crawling & Scraping (3-5 yrs)

Senior Software Engineer - IT (Python Developer with Web Crawling/Scraping, Selenium, MS SQL/MongoDB, Docker/Kubernetes)

Position: Senior Software Engineer - IT

Location: Hyderabad (Work from Office-3 Days)

Duration: 48 Months

Job Type: Contract

Work Type: Hyderabad (Work from Office-3 Days)

We are looking for a skilled and experienced Senior Software Engineer to join our IT team in Hyderabad (work from office 3 days a week). This is a 48-month contract position, where the candidate will work on key web development projects, primarily focusing on web crawling and scraping tasks.

You will be responsible for maintaining and enhancing our website by collecting outdated images, videos, and URLs, as well as ensuring the site reflects the latest product announcements.

You will also work in a collaborative environment, helping streamline processes and continuously improve our content management systems.

If you have a strong foundation in Python development, web crawling/scraping, and have worked with Docker/Kubernetes and Selenium, we want to hear from you.

Join us for this exciting opportunity where you will make a direct impact on the performance and content integrity of our website.

Key Responsibilities :

Web Crawling and Scraping :

- Utilize Python to develop and maintain web crawling and scraping solutions to gather and collect outdated images, videos, and URLs from different sources on the web.

- Efficiently extract content from websites, handle large-scale data extraction, and ensure the latest content is updated regularly on the platform.

- Develop and implement robust scraping systems that can work across multiple websites and platforms, ensuring reliability, scalability, and accuracy.

Content Management and Updates :

- Regularly upload new content, including updated images, videos, and product-related URLs to ensure the website remains current.

- Organize and validate the collected data to ensure it aligns with the latest announcements, ensuring that no outdated information remains accessible to users.

- Collaborate with content teams to verify and validate the content integration process, making sure that any updated product information or multimedia assets are correctly reflected.

Database Management and Data Storage :

- Use MS SQL Server or MongoDB to manage the collected data, ensuring it is stored efficiently and securely.

- Ensure data consistency and integrity when integrating new content into the website and the underlying systems.

- Troubleshoot database issues and optimize queries for faster data retrieval and better overall system performance.

Automation with Selenium :

- Work with Selenium for web testing and automated browsing to simulate user interactions and ensure the web pages are functioning as expected after updates.

- Automate content verification processes, ensuring the website works smoothly post-update, without any broken links or missing media files.

- Continuously improve and maintain test scripts to ensure consistent functionality across all platforms.

Containerization and Orchestration (Docker/Kubernetes) :

- Use Docker to containerize web scraping scripts and associated applications, ensuring easy deployment and execution in different environments.

- Leverage Kubernetes for orchestration, managing deployments, scaling, and maintaining web crawling processes.

- Ensure the infrastructure is scalable, reliable, and resilient, handling multiple crawling jobs simultaneously without significant performance issues.

Collaboration and Teamwork :

- Work closely with cross-functional teams including content managers, developers, and system administrators to streamline content management and web maintenance processes.

- Collaborate on system architecture and recommend improvements or new tools to improve the overall efficiency and accuracy of the scraping and crawling process.

- Share knowledge and provide mentorship to junior team members, offering guidance on best practices for web scraping, Python development, and containerization.

Testing and Quality Assurance :

- Perform unit testing, integration testing, and system testing on web scraping processes to ensure all functionalities are working as intended.

- Troubleshoot, identify, and resolve any issues related to crawling scripts or content updates on the website.

- Ensure that the content integration processes meet quality standards and comply with technical requirements.

Continuous Improvement and Optimization :

- Constantly monitor and optimize the performance of web crawling scripts and content management processes.

- Work on improving the speed and accuracy of content updates, reducing the time between crawling, scraping, and uploading of new media.

- Review system performance and suggest new technologies or improvements that could increase efficiency and reduce operational overhead.

Required Skills and Experience :

- Python Development (3 to 5 years of experience):

- Proficient in Python programming for web scraping and development.

- Experience with BeautifulSoup, Scrapy, Requests, or other relevant libraries for web crawling.

- Ability to develop robust, scalable, and maintainable Python scripts for web scraping and automation.

- Web Crawling & Scraping (3 to 5 years of experience):

- Extensive experience with web scraping technologies and techniques for data extraction from complex websites.

- Understanding of web scraping challenges, such as handling dynamic content, pagination, and site structure.

- Familiarity with handling CAPTCHAs, cookies, and other anti-scraping mechanisms.

Database Management (1 to 2 years of experience) :

- Experience with MS SQL Server or MongoDB for storing and managing data from web crawlers.

- Knowledge of data modeling, indexing, and query optimization for efficient data storage and retrieval.

- Selenium Automation (3 to 5 years of experience):

- Strong knowledge of Selenium for web automation and functional testing.

- Ability to write and execute Selenium scripts for web testing, content verification, and UI interaction simulation.

Containerization and Orchestration (1 to 2 years of experience) :

- Hands-on experience with Docker for containerizing applications and ensuring portability across different environments.

- Experience with Kubernetes for orchestration, scaling, and management of Docker containers.

Problem Solving and Debugging :

- Strong troubleshooting skills with an ability to resolve complex issues related to web crawling, content scraping, and integration.

- Ability to identify bottlenecks in the crawling process and optimize for efficiency.

Collaboration and Communication Skills :

- Strong communication skills with the ability to work in a team environment.

- Experience working in Agile development teams and collaborating with other team members to ensure timely and effective delivery of projects.

- Ability to write clear and concise technical documentation for processes, procedures, and troubleshooting.

Education and Qualifications :

- A Bachelor's or Master's Degree in Computer Science, Information Technology, or a related field.

- Certifications in Python Development, Docker, or Kubernetes are a plus.

Preferred Skills :

- Familiarity with cloud platforms such as AWS or Azure for hosting and scaling applications.

- Knowledge of CI/CD pipelines for automating deployment processes.

- Previous experience working with large-scale web scraping projects and content management systems.

Work Location :

Location : Hyderabad, India (Work from Office - 3 days a week)

Job Type :

- Contract (Duration: 48 months)