Add office photos

Thinkproject

Compare

based on 1 Review

4 Thinkproject Jobs

Senior Data Engineer

Thinkproject

based on 1 Review

5-10 years

Pune

1 vacancy

Senior Data Engineer

Thinkproject

posted 2mon ago

Flexible timing

Key skills for the job

+ 3 more

ETL Pipeline Development: Design, build, and maintain ETL pipelines using Matillion to extract data from SQL databases, transform it according to business rules, and load it into Snowflake.

Data Transformation in Snowflake: Write and manage transformations directly within Snowflake using Snowflake Tasks, Streams, and Stored Procedures to handle specific business logic or real-time data requirements.

Porting ETL to Snowflake & Spark: Migrate existing ETL pipelines from Matillion into Snowflake and Spark, utilizing the native features of both platforms to optimize data processing, storage, and analytics.

Collaborate with Stakeholders: Work closely with data architects, business analysts, and other teams to understand data requirements and ensure that data pipelines are aligned with business needs.

Performance Optimization: Continuously optimize ETL and data transformation processes in both Matillion and Snowflake for enhanced performance and reduced operational costs.

Monitoring & Troubleshooting: Monitor the health of data pipelines and Snowflake transformations, troubleshooting issues, and implementing solutions to ensure data accuracy and availability.

Data Preparation: Implement data cleansing, aggregation, and enrichment processes within Snowflake to prepare data for downstream reporting and analytics.

Snowflake Management: Manage and optimize Snowflakes data architecture, ensuring best practices in data modeling, partitioning, and indexing to support analytics and reporting.

Compliance & Governance: Ensure all ETL processes and transformations comply with data governance, privacy, and security requirements.

What you need to fulfill the role

Proficiency in Matillion: Extensive experience in building ETL pipelines and transformations using Matillion, extracting data from SQL databases, and loading it into Snowflake.

Expertise in Snowflake: Strong understanding of Snowflake architecture, including experience writing and managing Snowflake Data Warehouse, Snowflake Tasks, Streams, and Stored Procedures for real-time data processing and transformations. Additionally, experience optimizing Snowflake billing and credit consumption.

SQL Databases: Proven experience working with SQL databases (e.g., MySQL, PostgreSQL, Microsoft SQL Server) and writing complex SQL queries.

ETL Design & Optimization: Ability to design scalable ETL processes and optimize the existing ETL processes.

Data Transformation: Experience in data cleansing, aggregation, and transformation both in Matillion and directly within Snowflake.

Performance Tuning: In-depth knowledge of performance optimization techniques in both Matillion and Snowflake environments, ensuring efficient data processing.