Syren Technologies - Lead Data Engineer - ETL/Python (5-6 yrs)

About the Job :

As a Lead Data Engineer - ETL/Python, you will play a crucial role in interpreting complex data, analyzing results, and providing ongoing reports to improve business outcomes.

- You will design and implement high-performance data systems, manage large-scale data processing, and collaborate with various teams to meet business requirements.

- Additionally, you'll work extensively with cloud solutions and ensure data optimization, quality, and efficiency through the development of robust ETL pipelines.

- Analyze complex datasets and provide actionable insights using statistical techniques and advanced analytics.

- Continuously generate reports to track performance, trends, and opportunities for improvement.

- Develop, implement, and maintain databases, data collection systems, and data analytics tools.

- Optimize the efficiency and quality of existing systems by incorporating best practices and improving data management strategies.

-Responsible for designing, developing, and managing ETL (Extract, Transform, Load) pipelines.

- Build robust and scalable data solutions that handle large-scale data processing efficiently.

- Work with distributed systems and applications on the Azure Cloud platform, ensuring optimal performance, scalability, and security.

- Design and deploy cloud-based solutions and services to enhance data operations and business intelligence.

- Collaborate with business stakeholders, project managers, business analysts, and other IT teams to understand requirements and translate them into technical solutions.

- Engage with architects, developers, and modelers to ensure seamless data flow and integration.

- Gather and document business requirements and transform them into technical specifications.

- Perform data profiling, document source systems, and define transformation rules to be applied to the target systems or reports.

- Work on multiple projects, balancing competing priorities and shifting focus as needed throughout the workday.

- Adapt quickly to changing business needs and technical challenges.

- Strong programming skills in Scala, PySpark, and Python.

- Extensive knowledge of SQL and PLSQL, with a solid understanding of data modeling and database management.

- Proven experience working with ETL processes and data transformation.

- Hands-on experience with Azure Cloud Data Platform solutions.

- Ability to gather and elicit business requirements from stakeholders and translate them into actionable technical requirements.

- In-depth understanding of source systems and proficiency in data profiling to assess data quality and structure