Senior Data Engineer - Big Data Technologies (5-8 yrs)

Job Summary :

We are seeking a highly motivated and experienced Senior Data Engineer to join our team. This role requires a deep curiosity about our business and a passion for technology and innovation.

You will be responsible for designing and developing robust, scalable data engineering solution that drive our business intelligence and data-driven decision-making processes. If you thrive in a dynamic environment and have a strong desire to deliver top-notch data solutions, we want to hear from you.

Key Responsibilities :

- Collaborate with agile teams to design and develop cutting-edge data engineering solutions.

- Build and maintain distributed, low-latency, and reliable data pipelines ensuring high availability and timely delivery of data.

- Design and implement optimized data engineering solutions for Big Data workloads to handle increasing data volumes and complexities.

- Develop high-performance real-time data ingestion solutions for streaming workloads.

- Adhere to best practices and established design patterns across all data engineering initiatives.

- Ensure code quality through elegant design, efficient coding, and performance optimization.

- Focus on data quality and consistency by implementing monitoring processes and systems.

- Produce detailed design and test documentation, including Data Flow Diagrams,

- Technical Design Specs, and Source to Target Mapping documents.

- Perform data analysis to troubleshoot and resolve data-related issues.

- Automate data engineering pipelines and data validation processes to eliminate manual interventions.

- Implement data security and privacy measures, including access controls, key management, and encryption techniques.

- Stay updated on technology trends, experimenting with new tools, and educating team members.

- Collaborate with analytics and business teams to improve data models and enhance data accessibility.

- Communicate effectively with both technical and non-technical stakeholders.

Qualifications :

- Education : Bachelor's degree in Computer Science, Computer Engineering, or a related field.

- Experience : Minimum of 5+ years in architecting, designing, and building data engineering solutions and data platforms.

- Proven experience in building Lakehouse or Data Warehouses on platforms like Databricks or Snowflake.

- Expertise in designing and building highly optimized batch/streaming data pipelines using Databricks.

- Proficiency with data acquisition and transformation tools such as Fivetran and DBT.

- Strong experience in building efficient data engineering pipelines using Python and PySpark.

- Experience with distributed data processing frameworks such as Apache Hadoop, Apache Spark, or Flink.

- Familiarity with real-time data stream processing using tools like Apache Kafka, Kinesis, or Spark Structured Streaming.

- Experience with various AWS services, including S3, EC2, EMR, Lambda, RDS, DynamoDB, Redshift, and Glue Catalog.

- Expertise in advanced SQL programming and performance tuning.

Key Skills :

- Strong problem-solving abilities and perseverance in the face of ambiguity.

- Excellent emotional intelligence and interpersonal skills.

- Ability to build and maintain productive relationships with internal and external stakeholders.

- A self-starter mentality with a focus on growth and quick learning.

- Passion for operational products and creating outstanding employee experiences.