Coresight Research - Data Engineer - ETL/Data Warehousing (3-5 yrs)

Job Description :

As a Data Engineer, you will play a crucial role in designing, developing, and maintaining the data infrastructure and systems that enable efficient and accurate data collection, storage, processing, and analysis.

You will collaborate closely with data scientists, analysts, and other stakeholders to understand their requirements and implement robust solutions that address their data needs.

Responsibilities :

1. Data Pipeline Development :

- Design, develop, and maintain scalable and efficient data pipelines for collecting, processing, and storing large volumes of structured and unstructured data from various sources.

- Ensure data quality, reliability, and integrity throughout the pipeline.

2. Data Warehouse and Storage Management :

- Architect, implement, and manage data warehousing and storage solutions, including data lakes, data marts, and relational databases.

- Optimize data storage and retrieval mechanisms to support high-performance analytics and reporting.

3. Data Transformation and ETL :

- Transform raw data into usable formats by implementing Extract, Transform, Load (ETL) processes.

- Cleanse, filter, and aggregate data to ensure consistency and accuracy.

- Develop and maintain ETL workflows and schedules.

4. Data Modeling :

- Design and implement data models that support efficient data retrieval, analysis, and reporting.

- Collaborate with data scientists and analysts to understand their modeling requirements and provide them with structured datasets for analysis.

5. Web scraping :

- Design and deploy web scraping solutions to collect structured and unstructured data from websites, APIs, and other online repositories.

- Optimize web scraping processes by implementing efficient scraping strategies, managing API rate limits, handling dynamic content, and overcoming anti-scraping measures.

- Collaborate with cross-functional teams to identify relevant data sources, define data requirements, and establish scraping methodologies to acquire data in a reliable and automated manner.

6. Data Governance and Security :

- Establish data governance processes, policies, and standards to ensure data privacy, security, and compliance.

- Implement appropriate access controls and data protection measures.

Monitor and address data quality issues.

7. Performance Optimization :

- Identify and resolve performance bottlenecks in data processing and storage systems.

- Implement optimizations to improve query performance and reduce data processing times.

8. Collaboration and Documentation :

- Collaborate with cross-functional teams, including data scientists, analysts, software engineers, and business stakeholders, to understand their data needs and provide technical expertise.

- Document data engineering processes, systems, and configurations.

Requirements :

- Bachelor's or Master's degree in Computer Science, Engineering, or a related field.

- Proven experience as a Data Engineer or in a similar role, with a focus on data pipeline development, data warehousing, and ETL processes.

- Strong programming skills in languages such as Python & SQL

- Proficiency in working with databases (SQL and NoSQL) and data warehousing technologies (e.g. , Snowflake).

- Familiarity with cloud platforms (e.g. Azure, GCP) and their data-related services (e.g. , S3, EC2, BigQuery).

- Experience with data integration and ETL tools (e.g. , Apache Airflow, Informatica) is a plus.

- Solid understanding of data modeling concepts and database design principles.

- Strong problem-solving skills and the ability to analyze complex data-related issues.

- Excellent communication and collaboration skills, with the ability to work effectively in cross-functional teams.

- Attention to detail and a commitment to delivering high-quality, reliable data solutions