NCompas - Data Engineer - ETL/Python (4-8 yrs)

Job Title : Data Engineer.

Experience Level : 4+ Years.

Location : Hyderabad and Ahmedabad.

Key Responsibilities :

- Data Pipeline Development : Design, build, and maintain data pipelines to efficiently process and transform large datasets from various sources (databases, APIs, etc.).

- Data Integration : Integrate data from multiple sources, ensuring accurate and reliable data flow into data warehouses, data lakes, or other storage systems.

- Data Quality : Ensure the quality, consistency, and accuracy of data across systems, including monitoring data pipelines and troubleshooting issues as they arise.

- Database Management : Manage and optimize databases (SQL, NoSQL) for efficient storage, retrieval, and processing of data.

- ETL Processes : Develop and manage ETL (Extract, Transform, Load) processes to integrate data into the data warehouse or other storage solutions.

- Collaboration : Work closely with data scientists, analysts, and other cross-functional teams to understand data needs and ensure the availability of clean, reliable data for analysis and reporting.

- Performance Optimization : Optimize data storage, retrieval, and processing for speed and scalability.

- Documentation : Document data engineering processes, data pipeline workflows, and database schemas to ensure clarity and maintainability.

Who Are You?

- Experience : 4-8 years of experience as a Data Engineer or in a similar role working with data pipelines, ETL processes, and data warehousing.

- Programming Skills : Proficient in Python, SQL, and/or other programming languages for data engineering tasks.

- Database Expertise : Experience with relational databases (e.g., MySQL, PostgreSQL) and/or NoSQL databases (e.g., MongoDB, Cassandra).

- Data Tools : Hands-on experience with data engineering tools like Apache Kafka, Airflow, Spark, or Hive.

- Cloud Platforms : Experience with cloud-based data engineering tools and platforms (e.g., AWS, Google Cloud, Azure).

- Data Warehousing : Knowledge of data warehouse solutions (e.g., Redshift, BigQuery, Snowflake).

- ETL Process Expertise : Strong knowledge and experience in developing ETL pipelines and integrating multiple data sources.

- Problem-Solving : Strong troubleshooting skills with the ability to resolve data-related challenges and optimize data workflows.

- Collaboration Skills : Ability to collaborate effectively with data scientists, business analysts, and other teams to ensure data requirements are met.