Senior Data Engineer - Spark/Python (4-9 yrs)

Job Description :

We are looking for an experienced and innovative Senior Data Engineer to design, develop, and optimize scalable data systems and infrastructure.

The ideal candidate will play a key role in building robust data pipelines, ensuring the availability and quality of data, and supporting advanced analytics and machine learning initiatives.

This role requires a blend of technical expertise, leadership skills, and a strong commitment to best practices in data engineering.

Role & Responsibilities :

The core responsibilities for the job include the following :

- Design, implement, and maintain large-scale, distributed data platforms, including data lakes and warehouses.

- Develop scalable and secure data architectures to meet current and future business needs.

- Build, monitor, and optimize data pipelines to efficiently ingest, transform, and store data from various sources.

- Automate repetitive data engineering tasks to enhance productivity and consistency.

- Work closely with data scientists, analysts, and business stakeholders to understand data requirements and deliver solutions.

- Collaborate with software engineers to integrate data systems with applications and services.

- Ensure high performance, scalability, and availability of data infrastructure.

- Optimize queries, workflows, and storage solutions for efficiency and cost-effectiveness.

- Implement and maintain data quality, governance, and security best practices.

- Enforce data standards and compliance with regulations (e., GDPR, HIPAA).

- Provide technical guidance and mentorship to junior engineers and team members.

- Lead technical projects and contribute to strategic decision-making related to data systems.

- Stay updated on emerging technologies and trends in data engineering.

- Evaluate and recommend new tools, frameworks, and methodologies to improve processes and systems.

Requirements :

- Bachelor's or master's degree in computer science, Engineering, Information Systems, or a related field.

- 4+ years of experience in data engineering, big data systems, or a related field.

- Proven experience in architecting and managing large-scale data solutions.

Technical Skills :

- Proficiency in Python, Java, or Scala for data processing.

- Expertise in Apache Spark, Hadoop, Kafka, or Flink.

- Hands-on experience with AWS (e., S3 Redshift, EMR), Azure, or Google Cloud (e., BigQuery, Dataflow).

- Strong knowledge of Snowflake, Redshift, BigQuery, or similar platforms.

- Experience with Apache Airflow, DBT, Informatica, or similar tools.

- Advanced skills in SQL and NoSQL databases (e., PostgreSQL, Cassandra, MongoDB).

- Familiarity with CI/CD pipelines, Docker, Kubernetes, and infrastructure as code (e., Terraform)