Principal Data Engineer - ETL Development

Job Responsibilities :

1. ETL Development and Data Processing :

- Design, develop, and implement ETL processes to extract, transform, and load data from various sources into data warehouses or analytics systems.

- Maintain and enhance ETL pipelines to ensure data quality, integrity, and consistency.

2. DevOps and CI/CD Pipeline Management :

- Implement and support CI/CD pipelines for Enterprise Data Warehouse (EDW) code development and deployment using GitLab.

- Manage and configure workflows for code repositories, ensuring seamless code integration and promotion across environments.

3. Data Analysis and Reporting :

- Apply statistical analysis and data mining techniques to uncover trends, patterns, and insights within large datasets.

- Develop and maintain dashboards, reports, and visualizations to communicate key insights to stakeholders.

4. Programming and Scripting :

- Develop and maintain Linux shell scripting, Python, and PL/SQL code for data extraction, transformation, and automation tasks.

- Create efficient scripts for data replication, migration, and data integration between systems.

5. Quality Assurance and System Optimization :

- Perform quality assurance checks on system installations, configurations, and code to ensure performance and scalability.

- Recommend improvements to optimize performance and reliability of data processes and systems.

6. Business Understanding and Translation :

- Work closely with business stakeholders to understand their goals, processes, and requirements, translating them into technical data solutions.

- Ensure alignment between technical solutions and business objectives to drive strategic decision-making.

7. Data Governance and Security :

- Implement data governance best practices to ensure the security, integrity, and privacy of enterprise data.

- Adhere to data security regulations and policies, ensuring that data is handled with confidentiality and compliance.

8. Collaboration and Stakeholder Communication :

- Collaborate with cross-functional teams, including data analysts, engineers, and business teams, to deliver impactful data solutions.

- Present data findings and actionable insights to stakeholders at various organizational levels, both written and verbally.

Required Qualifications :

Experience :

- 8+ years of experience in IT with a focus on Business Intelligence, Data Warehousing, and Analytics systems.

- Proven experience with DevOps methodologies, CI/CD pipelines, and GitLab workflows.

- Proficient in ETL development and creating data workflows in SQL, PL/SQL, Python, and Linux shell scripting.

Technical Skills :

- Strong knowledge of open-source technologies, Linux, and GitLab for codebase management.

- Experience in data replication and data migration processes.

- Hands-on experience with Python and Node.js for programming and automation tasks.

Soft Skills :

- Excellent problem-solving and analytical abilities.

- Strong communication skills with the ability to convey complex technical concepts to non-technical stakeholders.

- Ability to manage multiple projects in a dynamic and fast-paced environment.

Preferred Skills :

- Experience with data governance principles and ensuring data security compliance.

- Familiarity with data quality assurance and optimization techniques.

- Understanding of modern data platforms such as cloud-based data warehouses and data lakes.

Location - Hyderabad,Ahmedabad,pune,chennai,kolkata.