Hindustan Times - Software Development Engineer II/III - Python (2-7 yrs)

Overview:

We are seeking talented Software Developers proficient in Python, with a specialisation in data engineering and web crawling. This role entails the design and implementation of resilient data solutions, coupled with the development of efficient web crawlers for extracting information from diverse online sources. The ideal candidate will exhibit a passion for utilising open-source libraries to gather, process, and analyse data.

Key responsibilities include :

- Collaborate with product managers, cross-functional teams, and stakeholders to comprehend data requirements. Translate requirements into technical specifications, ensuring data quality and integrity

- Design and implement data engineering solutions using Python to manage large and diverse datasets.

- Ensure efficient storage, retrieval, and processing of data, following best practices

- Develop and maintain web crawlers in Python for extracting data from diverse online platforms

- Create scripts for data cleaning, transformation, and normalisation, ensuring high-quality data

- Design and implement data models, databases, and processing components for optimal performance

- Integrate with external APIs to enrich data and enhance overall information retrieval processes

- Establish monitoring tools to track the health and performance of web crawlers and data processing systems. Identify and address performance bottlenecks

- Develop and execute comprehensive test plans for features and data-driven functionalities

- Stay updated with emerging technologies and contribute to the continuous improvement of development processes

Functional Competencies :

- B.E./B.Tech. in CS/IT or a related field with 2+ years of hands-on experience as a software developer

- Expertise in data engineering, encompassing efficient data storage, processing, transformation, and retrieval of large datasets

- Strong understanding of Python, MongoDB, MySQL, Elasticsearch, Redis, and related technologies

- Proficiency in web crawling using Python libraries and frameworks such as Beautiful Soup, Scrapy, Pandas, NumPy

- Experience in Restful API development and integration.

- Familiarity with cloud platforms, such as AWS and Azure.

- Strong problem-solving skills with meticulous attention to detail

- Ability to work collaboratively in a dynamic development environment