Data Engineer

We are seeking a skilled Data Engineer with at least 5 years of experience to join our data analytics team, focusing on building robust data pipelines and systems to support the creation of dynamic dashboards. The role involves designing, building, and optimizing data architecture, enabling real-time data flow for visualization and analytics. The Data Engineer will be responsible for managing ETL processes, ensuring data quality, and supporting the scalable integration of various data sources into our analytics platform.
The ideal candidate should have extensive experience in working with complex data architectures, managing ETL workflows, and ensuring seamless data integration across platforms. They should also have a deep understanding of cloud technologies and database management.

Key Responsibilities:
Data Pipeline Development
o Design, build, and maintain scalable ETL (Extract, Transform, Load) processes for collecting, storing, and processing structured and unstructured data from multiple sources.
o Develop workflows to automate data extraction from APIs, databases, and external sources.
o Ensure data pipelines are optimized for performance and handle large data volumes with minimal latency.
Data Integration and Management
o Integrate data from various sources (e.g., databases, APIs, cloud storage) into the centralized daIta warehouse or data lake to support real-time dashboards.
o Ensure smooth data flow and seamless integration with analytics tools like Power BI and Tableau.
o Manage and maintain data storage solutions, including relational (SQL-based) and NoSQL databases.

Data Quality and Governance
o Implement data validation checks and quality assurance processes to ensure data accuracy, consistency, and integrity.
o Develop monitoring systems to identify and troubleshoot data inconsistencies, duplications, or errors during ingestion and processing.
o Ensure compliance with data governance policies and standards, including data protection regulations such as the Digital Personal Data Protection (DPDP) Act.
Database Management and Optimization
o Design and manage both relational and NoSQL databases, ensuring efficient storage, query performance, and reliability.
o Optimize database performance, ensuring fast query execution times and efficient data retrieval for dashboard visualization.
o Implement data partitioning, indexing, and replication strategies to support large-scale data operations.
Data Security and Compliance
o Ensure that all data processes adhere to security best practices, including encryption, authentication, and access control.
o Implement mechanisms for secure data storage and transmission, especially for sensitive government or public sector data.
o Conduct regular audits of data pipelines and storage systems to ensure compliance with relevant data protection regulations.
Cloud Infrastructure and Deployment

o Deploy and manage cloud-based data solutions using AWS, Azure, or GCP, including data lakes, data warehouses, and cloud-native ETL tools.
o Set up cloud infrastructure to support high availability, fault tolerance, and scalability of data systems.
o Monitor cloud usage and optimize costs for data storage, processing, and retrieval.
Performance Monitoring and Troubleshooting
o Continuously monitor data pipeline performance and data ingestion times to identify bottlenecks and areas for improvement

Troubleshoot and resolve any data flow issues, ensuring high availability and reliability of data for dashboards and analytics.
o Implement logging and alerting mechanisms to detect and address any operational issues proactively.
Qualifications:
Education: Bachelor s degree in Computer Science, Information Technology, Data Engineering, or a related field. A Master s degree is a plus.
Experience: At least 5 years of hands-on experience as a Data Engineer, preferably in a data analytics or dashboarding environment.