BrightEdge - Software Development Engineer III - Web Scraping (3-6 yrs)

We are seeking a Backend Engineer to design, implement, and optimize data collection, storage, and processing pipelines. The ideal candidate will have expertise in handling structured and unstructured data, optimizing query performance, and ensuring efficient data retrieval. This role involves working with APIs, cloud logging, data streaming, and anomaly detection to maintain robust data workflows.

Responsibilities :

- Develop and maintain data collection pipelines to efficiently ingest and store structured and unstructured data.

- Implemented web scraping techniques and APIs for real-time and batch data retrieval.

- Optimize data storage and query performance in MySQL, Redis, and other databases.

- Ensure efficient logging (cloud & local) and real-time data streaming.

- Implement alerts and anomaly detection for monitoring data inconsistencies.

- Work on data optimization strategies for faster retrieval and efficient storage.

- Design and maintain scalable, fault-tolerant data architectures.

- Collaborate with cross-functional teams to ensure data integrity and performance.

Requirements :

- Proficiency in Web Scraping and API Integration for data collection.

- Experience in Cloud Logging, Data Logging, and Data Streaming.

- Strong knowledge of MySQL, Redis, and other storage solutions.

- Expertise in data optimization, query optimization, and efficient retrieval mechanisms.

- Hands-on experience in structured and unstructured data handling.

- Ability to design and implement data pipelines.

- Experience in implementing scripts, alerts, and anomaly detection mechanisms.

- Strong problem-solving skills with a focus on performance and scalability.

- Experience with cloud platforms (AWS, GCP, or Azure).

- Knowledge of Big Data frameworks (Kafka, Spark, etc. ).

- Familiarity with Python, SQL, or other scripting languages.