Senior Data Engineer - Apache NiFi (8-11 yrs)

Key Responsibilities :

Data Pipeline Development :

- Design, develop, and maintain complex data pipelines using Apache NiFi to ingest, process, and transform large volumes of data.

- Optimize data flows for performance, scalability, and reliability.

- Implement data validation and error handling mechanisms to ensure data quality and integrity.

Data Integration :

- Integrate data from various sources, including databases, APIs, and filesystems, into the data warehouse.

- Develop ETL processes to extract, transform, and load data into target systems.

- Optimize ETL processes for performance and efficiency.

Cloud Technologies :

- Leverage AWS services (e., S3, EMR, Lambda) to build scalable and cost-effective data solutions.

- Configure and manage AWS infrastructure for data pipelines.

- Implement security best practices to protect sensitive data.

Data Modeling and Warehousing :

- Collaborate with data analysts and business stakeholders to understand data requirements.

- Design and implement data models for data warehouses and data marts.

- Optimize data warehouse performance and query execution.

Problem-Solving and Troubleshooting :

- Identify and resolve data quality issues and pipeline failures.

- Analyze system logs and performance metrics to identify bottlenecks and optimization opportunities.

- Implement monitoring and alerting mechanisms to proactively address issues.

Collaboration :

- Work closely with data engineers, data analysts, and other team members to deliver high-quality solutions.

- Participate in code reviews and knowledge sharing sessions.

- Contribute to a positive and collaborative team culture.

Required Skills and Experience :

- 8+ years of experience in data engineering and ETL processes.

- Strong proficiency in Apache NiFi, including data flow design, processor configuration, and performance tuning.

- Expertise in SQL, particularly PostgreSQL.

- In-depth knowledge of AWS services, including S3, EMR, Lambda, and Glue.

- Experience with data warehousing concepts and methodologies.

- Understanding of data modeling and data quality best practices.

- Strong problem-solving and troubleshooting skills.

- Excellent communication and collaboration skills.

- Ability to work independently and as part of a team.

Preferred Skills :

- Experience with other data integration tools (e., Talend, Informatica).

- Knowledge of scripting languages (e., Python, Shell).

- Familiarity with cloud-native data platforms (e., Databricks, Snowflake).

- Experience with data security and privacy best practices