Spigot Software - Senior Data Engineer - ETL/Python (5-12 yrs)

Job Description :

We are seeking experienced, motivated and enthusiastic Data Engineering Specialists to join our team. As part of the team, you will be involved in the implementation of the ongoing and new initiatives for our company and contribute to our mission of delivering high-quality data-driven solutions to our clients.

Primary Skills : Azure functions/AWS Lambda, Python, SQL, Snowflake, FiveTran, ELT/ETL

Secondary Skills : Azure OCR, API Development, DBT

Role Description :

The data engineering role requires creating and managing technical solutions on the modern data stack (Azure, Snowflake, fivetran), architecting, building, and managing data flows/pipelines and construct data models within Snowflake to be used for other downstream processes or for analysis purposes.

Role Responsibility :

- Design, develop, and maintain scalable data pipelines and analytics solutions

- Collaborate with stakeholders to gather requirements and translate business needs into technical solutions

- Develop efficient code with unit testing and code documentation

- Ensuring accuracy and integrity of data and applications through analysis, coding, documenting, testing, and problem solving

- Optimize and fine-tune data models, SQL queries, and transformations for performance and scalability

- Design, develop, and maintain scalable data models and transformations in conjunction with Snowflake, ensure the effective transformation and load data from diverse sources into data warehouse or data lake.

- Integrate FiveTran connectors to streamline data ingestion from various sources into Snowflake

- Develop custom Python scripts and functions to automate data workflows and enhance system capabilities

- Collaborate with data engineers to automate and orchestrate data pipelines.

- Provide technical expertise and support to resolve data-related issues and troubleshoot system failures

- Collaborate with the API development team to integrate data pipelines with external systems and applications

- Implement techniques for query optimization, caching, and workload management.

- Perform regular audits and troubleshooting to maintain system performance.

- Implement and manage data security measures, including access controls, encryption, and data masking.

- Ensure compliance with data governance policies and regulatory requirements.

- Communicate with all the project stakeholders on the project status

- Lead and participate in initiatives to enhance data platform capabilities and performance.

- Stay updated on emerging trends and technologies in data engineering, cloud computing, and analytics domains

Role Requirements :

- Bachelor's degree in Computer Science, Engineering, or a related field

- Proven experience of at least 5 years as a data engineer, ETL developer, or similar role, with a focus on Python and Snowflake

- Strong proficiency in SQL and database concepts, with hands-on experience in Snowflake data warehouse

- Strong proficiency in programming languages such as Python for data manipulation, automation, and scripting. Expertise in Azure durable functions

- Knowledgeable in relational databases, nonrelational databases, data pipelines (ELT/ETL), and file stores

- Knowledgeable in performance tuning and optimization

- Experience with cloud platforms like Azure and tools like Azure OCR for data extraction and processing

- Familiarity with data integration tools like FiveTran

- Knowledge of API development principles and experience integrating data pipelines with external systems

- Familiarity with CI/CD pipelines and version control systems (e.g., Git).

- Proficient in written, verbal and presentation communication (English)

- Ability to work effectively in a fast-paced, collaborative environment and manage multiple priorities

- Excellent analytical, problem-solving, and communication skills

- Ability to work effectively in a fast-paced, collaborative environment and manage multiple priorities

- Snowflake certification (SnowPro Core, SnowPro Advanced)