DynPro - Senior Data Engineer - ETL/Scala/Python (5-9 yrs)

A Typical Day :

- Design, build and maintain robust and efficient data pipelines to support various produc and business needs.

- Design data models for efficient storage and retrieval to meet critical product a business requirements.

- Build scalable data pipelines (SparkSQL & Scala) leveraging Airflow scheduler/execut framework.

- Implement SQL based data quality checks and investigate data issues.

- Collaborate with cross-functional teams, including product managers, engineers, and business partners, to align on data requirements and develop scalable systems.

Your Expertise :

- 5-9+ years of relevant industry experience with a BS/Masters, or 2+ years with a PhD.

- Extensive experience designing, building, and operating robust distributed data platforms (e.g , Spark, Kafka, Flink, HBase) and handling data at the petabyte scale.

- Strong knowledge of Java, Scala, or Python, and expertise with data processing technologies and query authoring (SQL).

- Demonstrated ability to analyze large data sets to identify gaps and inconsistencies, provide data insights, and advance effective product solutions.

- Expertise with ETL schedulers such as Apache Airflow, Luigi, Oozie, AWS Glue similar frameworks.

- Solid understanding of data warehousing concepts and hands-on experience with relational databases (e.g , PostgreSQL, MySQL) and columnar databases (e.g , Redshift, BigQuery, HBase, ClickHouse).

- Excellent written and verbal communication skills.

Responsibilities :

- Design, develop, and implement data pipelines and solutions

- Work closely with data scientists and analysts to understand their data needs

- Develop and maintain data quality standards Automate data processing tasks Monitor and troubleshoot data pipelines