Senior Data Engineer - ETL/Data Pipeline (8-10 yrs)

Job Purpose :

We are seeking a highly skilled and experienced Senior Data Engineer to join our growing data team. This role will be instrumental in designing, developing, and optimizing scalable data pipelines and warehousing solutions that support business intelligence, analytics, and machine learning initiatives.

The ideal candidate will have a deep understanding of data engineering principles, cloud platforms, and hands-on experience in real-time data streaming, data integration, and performance optimization. This position requires close collaboration with data scientists, analysts, and business stakeholders to deliver high-quality data solutions that drive strategic decision-making.

Roles and Responsibilities :

Data Pipeline Development :

- Design, develop, and maintain scalable and efficient data pipelines using ETL/ELT tools.

- Implement and optimize real-time data streaming solutions using Kafka, Kinesis, or similar technologies.

Data Warehousing & Infrastructure :

- Architect, implement, and optimize data warehousing solutions (e.g., Snowflake, Redshift, BigQuery).

- Design and implement dimensional modeling and data schema best practices.

- Leverage cloud platforms (AWS, Azure, GCP) to build and manage scalable, cost-efficient data infrastructure.

Data Integration & Processing :

- Integrate data from multiple sources, including relational databases, NoSQL databases, APIs, and streaming platforms.

- Develop and maintain logical and physical data models that meet business requirements.

- Ensure data governance, security, and compliance with industry regulations (e.g., GDPR, HIPAA).

Performance Optimization & Quality Assurance :

- Analyze and optimize query performance for large-scale datasets.

- Implement data validation checks and automated monitoring for data quality and consistency.

- Troubleshoot and resolve performance bottlenecks in data workflows.

Collaboration & Mentorship :

- Work closely with data scientists, analysts, and business leaders to translate data needs into actionable solutions.

- Provide technical mentorship to junior data engineers, promoting best practices and continuous learning.

- Maintain comprehensive technical documentation for data pipelines, infrastructure, and processes.

Required Skills :

- Cloud Platforms : Strong expertise in at least one major cloud provider (AWS, Azure, or GCP) and its data engineering services (S3, Redshift, Glue, Azure Data Lake, Azure Synapse, BigQuery, Dataflow).

- Data Warehousing : Expertise in dimensional modeling, star schema, and snowflake schema.

- ETL/ELT : Proficiency in tools such as Apache Spark, Apache Airflow, Informatica PowerCenter, Matillion.

- Programming & Scripting : Strong skills in Python, Java, or Scala for data processing.

- Databases : Hands-on experience with relational (PostgreSQL, MySQL, SQL Server) and NoSQL databases (MongoDB, Cassandra).

- Big Data Technologies : Experience with Hadoop, Spark for large-scale data processing.

- SQL & Query Optimization : Ability to write and optimize complex SQL queries and stored procedures.

- Version Control : Proficiency with Git and CI/CD workflows.

- Operating Systems : Strong understanding of Linux environments.

Preferred Skills :

- Knowledge of data governance and data quality frameworks.

- Experience with machine learning and data science concepts.

- Familiarity with containerization and orchestration (Docker, Kubernetes).

- Understanding of DevOps practices and infrastructure as code.

- Hands-on experience with data visualization tools (Tableau, Power BI).

Required Competencies :

- Analytical Thinking : Ability to analyze complex datasets and optimize workflows.

- Problem-Solving : Strong troubleshooting skills for data-related challenges.

- Communication & Collaboration : Ability to convey technical concepts to non-technical stakeholders.

- Adaptability : Willingness to learn and implement new technologies in a rapidly evolving data landscape.

- Attention to Detail : Commitment to ensuring data accuracy, security, and compliance.

Required Qualifications :

- Bachelor's degree in Computer Science, Information Technology, or a related field.

- 8+ years of experience in IT, with significant focus on data engineering.