Welspun GCC - Lead Data Engineer (3-6 yrs)

Responsibilities of Data Engineering :

- Lead and manage the data engineering team, ensuring high performance and smooth workflow.

- Design, construct, install, test, and maintain highly scalable data management systems.

- Develop and implement data pipelines, ensuring robustness and data quality.

- Collaborate with data scientists and architects on several projects.

- Use Azure Databricks for big data analytics and processing.

- Implement ETL processes and work closely with the DevOps team to automate them.

- Use SQL and Python for data querying and analysis.

- Implement data visualization tools and techniques to present data in a way that is easily understandable for non-technical team members.

- Use Microsoft Fabric for data integration and processing.

- Apply strong analytical thinking to interpret data and make strategic decisions.

- Ensure data privacy and comply with all relevant data protection regulations.

- Continually improve ongoing reporting and analysis processes, automating or simplifying self-service support for secondary datasets.

- Provide technical expertise and recommendations in assessing new IT software projects and initiatives to support and enhance our existing systems.

- Conduct team meetings and provide training to team members as necessary.

- Stay updated with the latest industry trends and technologies to ensure the company's data infrastructure is up-to-date.

- The Data Engineering Lead should be a strategic thinker, excellent team leader, and a problem-solver with a strong analytical mindset.

- The ability to manage and interpret large amounts of data is crucial for this role.

Functional Competencies :

Data Engineering : The candidate should have a strong understanding of database structures, theories, principles, and practices. They should be able to design, construct, install, test and maintain highly scalable data management systems.

Data Visualization : The candidate should be proficient in transforming data into readable, goal driven reports for continued innovation and growth. They should be able to use data visualization tools to present data in a clear, concise manner.

SQL : The candidate should have a strong understanding of SQL database coding and management. They should be able to write and optimize SQL statements for data access and retention.

Python : The candidate should have experience in using Python for creating high performing data processing scripts. They should be proficient in Python libraries for data analysis.

Azure Databricks : The candidate should have experience in using Azure Databricks for big data analytics and artificial intelligence solutions.

DevOps : The candidate should have a good understanding of DevOps principles and tools, and should be able to work in a fastpaced environment with a focus on continuous delivery.

Data Pipeline : The candidate should have experience in building and maintaining reliable and scalable data pipelines to meet business needs.

ETL : The candidate should have experience in designing, building, and maintaining ETL pipelines.

Behavioural Competencies :

- Team Management : The candidate should have strong leadership skills and the ability to manage a team effectively. They should be able to delegate tasks, motivate team members, and resolve conflicts.

- Analytical Thinking : The candidate should have strong analytical skills and the ability to make data driven decisions. They should be able to analyze complex data sets and derive meaningful insights.

- Data Analysis : The candidate should have strong data analysis skills.

They should be able to analyze large amounts of raw information to find patterns and use datadriven insights to help the company make business decisions.

Good to have skills :

- Microsoft Fabric : The candidate should have a deep understanding of Microsoft Fabric and its use in building scalable and reliable applications. They should be able to use Microsoft Fabric to design and implement highquality solutions