Snowflake Data Engineer

Position Name-Snowflake Data Engineer
Experience: 3+ years

Key Skills: Snowflake, Kafka, ETL, SQL, Data Pipeline, Python, Knowledge in Power BI

We are looking for a skilled Data Engineer with expertise in Snowflake and Apache Kafka to contribute to a technology-driven company specializing in innovative data solutions. This role involves designing, developing, and optimizing data pipelines to build a scalable and high-performance data ecosystem. The ideal candidate will be responsible for integrating various data sources, ensuring seamless data flow, and collaborating with cross-functional teams to support advanced analytics initiatives.

What you'll be doing:

• Develop, implement, and manage scalable Snowflake data warehouse solutions using advanced features such as materialized views, task automation, and clustering.
• Design and build real-time data pipelines from Kafka and other sources into Snowflake using Kafka Connect, Snowpipe, or custom solutions for streaming data ingestion.
• Create and optimize ETL/ELT workflows using tools like DBT, Airflow, or cloud-native solutions to ensure efficient data processing and transformation.
• Tune query performance, warehouse sizing, and pipeline efficiency by utilizing Snowflakes Query Profiling, Resource Monitors, and other diagnostic tools.
• Work closely with architects, data analysts, and data scientists to translate complex business requirements into scalable technical solutions.
• Enforce data governance and security standards, including data masking, encryption, and RBAC, to meet organizational compliance requirements.
• Continuously monitor data pipelines, address performance bottlenecks, and troubleshoot issues using monitoring frameworks such as Prometheus, Grafana, or Snowflake-native tools.
• Provide technical leadership, guidance, and code reviews for junior engineers, ensuring best practices in Snowflake and Kafka development are followed.
• Research emerging tools, frameworks, and methodologies in data engineering and integrate relevant technologies into the data stack.

What you need:

Basic Skills:

• 3+ years of hands-on experience with Snowflake data platform, including data modeling, performance tuning, and optimization.
• Strong experience with Apache Kafka for stream processing and real-time data integration.
• Proficiency in SQL and ETL/ELT processes.
• Solid understanding of cloud platforms such as AWS, Azure, or Google Cloud.
• Experience with scripting languages like Python, Shell, or similar for automation and data integration tasks.
• Familiarity with tools like dbt, Airflow, or similar orchestration platforms.
• Knowledge of data governance, security, and compliance best practices.
• Strong analytical and problem-solving skills with the ability to troubleshoot complex data issues.
• Ability to work in a collaborative team environment and communicate effectively with cross-functional teams

Responsibilities:

• Design, develop, and maintain Snowflake data warehouse solutions, leveraging advanced Snowflake features like clustering, partitioning, materialized views, and time travel to optimize performance, scalability, and data reliability.
• Architect and optimize ETL/ELT pipelines using tools such as Apache Airflow, DBT, or custom scripts, to ingest, transform, and load data into Snowflake from sources like Apache Kafka and other streaming/batch platforms.
• Work in collaboration with data architects, analysts, and data scientists to gather and translate complex business requirements into robust, scalable technical designs and implementations.
• Design and implement Apache Kafka-based real-time messaging systems to efficiently stream structured and semi-structured data into Snowflake, using Kafka Connect, KSQL, and Snow pipe for real-time ingestion.
• Monitor and resolve performance bottlenecks in queries, pipelines, and warehouse configurations using tools like Query Profile, Resource Monitors, and Task Performance Views.
• Implement automated data validation frameworks to ensure high-quality, reliable data throughout the ingestion and transformation lifecycle.
• Pipeline Monitoring and Optimization: Deploy and maintain pipeline monitoring solutions using Prometheus, Grafana, or cloud-native tools, ensuring efficient data flow, scalability, and cost-effective operations.
• Implement and enforce data governance policies, including role-based access control (RBAC), data masking, and auditing to meet compliance standards and safeguard sensitive information.
• Provide hands-on technical mentorship to junior data engineers, ensuring adherence to coding standards, design principles, and best practices in Snowflake, Kafka, and cloud data engineering.
• Stay current with advancements in Snowflake, Kafka, cloud services (AWS, Azure, GCP), and data engineering trends, and proactively apply new tools and methodologies to enhance the data platform.