Data Engineer - ETL/Python (5-7 yrs)

Job Description : Data Engineer (Python).

About Us :

VCCEdge, part of Mosaic Digital is an upcoming company dedicated to building innovative products for private market stakeholders.

Our platform provides valuable insights, analytics, and solutions to empower decision-makers to navigate the dynamic landscape of the private market.

Overview: We are seeking an experienced Data Engineer proficient in Python to lead and optimize data engineering solutions.

This role involves architecting, designing, and implementing scalable, high-performance data pipelines and web crawling solutions to extract, process, and analyze complex datasets.

The ideal candidate will bring more than 4 years of experience, demonstrating expertise in managing large-scale data ecosystems, automation, and cloud-based data platforms.

Responsibilities :

- Architect and develop scalable data engineering solutions to manage, process, and transform large and diverse datasets efficiently.

- Design and implement robust data pipelines using Python to ensure high availability, integrity, and quality of data.

- Lead the development of web crawlers using Scrapy, BeautifulSoup, and other open-source frameworks for extracting data from various online sources.

- Develop and optimize ETL/ELT pipelines, ensuring efficiency and reliability.

- Integrate and manage data storage solutions, including MongoDB, MySQL, Elasticsearch, and Redis.

- Implement and enhance automation scripts to streamline data collection, transformation, and normalization.

- Monitor and optimize data processing workflows, identifying and resolving performance bottlenecks.

- Ensure security, scalability, and compliance in data architecture and workflows.

- Collaborate with cross-functional teams to integrate data insights into larger systems and products.

- Develop and maintain APIs to facilitate data access, retrieval, and integration.

- Lead data governance initiatives to maintain data consistency, quality, and accessibility.

- Stay updated with the latest advancements in big data technologies, cloud computing (AWS/Azure), and data engineering best practices.

Requirements :

- Bachelor's or Master's degree in Computer Science, Engineering, or a related field.

- 4+ years of experience as a Data Engineer with a strong focus on Python.

- Expertise in data engineering, ETL pipelines, data warehousing, and distributed systems.

- Proven experience in designing high-performance databases and optimized data storage solutions.

- Strong proficiency in web crawling and data extraction using Python libraries like Scrapy, BeautifulSoup, Selenium.

- Experience with cloud platforms (AWS, Azure, or GCP) for data storage, processing, and analytics.

- Solid understanding of SQL, NoSQL databases, and indexing strategies.

- Experience with data processing frameworks such as Apache Spark, Kafka, or Airflow is a plus.

- Strong problem-solving abilities and a detail-oriented approach to data architecture.

- Ability to work independently and collaboratively in a fast-paced environment.