Data Engineer - Senior

GPP Database Link

Job Summary:

Leads projects for the design, development, and maintenance of a data and analytics platform. Effectively and efficiently processes, stores, and makes data available to analysts and other consumers. Works with key business stakeholders, IT experts, and subject-matter experts to plan, design, and deliver optimal analytics and data science solutions. Works on one or many product teams at a time. Though the role category is generally listed as Remote, this specific position is designated as Hybrid.

Key Responsibilities:

• Designs and automates deployment of distributed systems for ingesting and transforming data from various sources (relational, event-based, unstructured).
• Designs and implements frameworks to continuously monitor and troubleshoot data quality and integrity issues.
• Implements data governance processes, including metadata management, access control, and retention policies for internal and external users.
• Provides guidance on building reliable, efficient, scalable, and quality data pipelines with monitoring and alert mechanisms that combine a variety of sources using ETL/ELT tools or scripting languages.
• Designs and implements physical data models to define database structures and optimize database performance through indexing and table relationships.
• Participates in optimizing, testing, and troubleshooting data pipelines.
• Develops and operates large-scale data storage and processing solutions using distributed and cloud-based platforms (e. g. , Data Lakes, Hadoop, HBase, Cassandra, MongoDB, Accumulo, DynamoDB).
• Uses innovative tools, techniques, and architectures to automate common data preparation and integration tasks, minimizing manual and error-prone processes.
• Assists in renovating data management infrastructure to drive automation in data integration and management.
• Ensures the timeliness and success of critical analytics initiatives by using agile development methodologies such as DevOps, Scrum, and Kanban.
• Coaches and develops less experienced team members.

Preferred Experience:

• Intermediate experience in a relevant discipline is required.
• Knowledge of the latest technologies and trends in data engineering is highly preferred.
• Familiarity with analyzing complex business systems, industry requirements, and data regulations.
• Background in processing and managing large datasets.
• Design and development experience for Big Data platforms using open-source and third-party tools.
• Experience with SPARK, Scala/Java, Map-Reduce, Hive, HBase, and Kafka.
• Hands-on experience with SQL query language.
• Experience in clustered compute cloud-based implementations.
• Experience developing applications requiring large file movement in a cloud-based environment and using various data extraction tools.
• Experience in building analytical solutions.
• Exposure to IoT technology.
• Experience in Agile software development methodologies.

Technical Skills:

• Programming Languages: Proficiency in Python, Java, and/or Scala.
• Database Management: Expertise in SQL and NoSQL databases.
• Big Data Technologies: Hands-on experience with Hadoop, Spark, Kafka, and similar frameworks.
• Cloud Services: Experience with Azure, Databricks, and AWS platforms.
• ETL Processes: Strong understanding of Extract, Transform, Load (ETL) processes.
• Data Replication: Working knowledge of replication technologies like Qlik Replicate is a plus.
• API Integration: Experience working with APIs to consume data from ERP and CRM systems.

Education, Licenses, and Certifications:

• Bachelors degree in a relevant technical discipline, or equivalent experience required.
• This position may require licensing for compliance with export controls or sanctions regulations.

Competencies:

• System Requirements Engineering: Translates stakeholder needs into verifiable requirements, tracks status, and assesses impact changes.
• Collaborates: Builds partnerships and works collaboratively with others to meet shared objectives.
• Communicates Effectively: Delivers multi-mode communications tailored to different audiences.
• Customer Focus: Builds strong customer relationships and provides customer-centric solutions.
• Decision Quality: Makes good and timely decisions that drive the organization forward.
• Data Extraction: Performs ETL activities from various sources using appropriate tools and technologies.
• Programming: Develops, tests, and maintains code using industry standards, version control, and automation tools.
• Quality Assurance Metrics: Measures and assesses solution effectiveness using IT Operating Model (ITOM) standards.
• Solution Documentation: Documents knowledge gained and communicates solutions for improved productivity.
• Solution Validation Testing: Validates configurations and solutions to meet customer requirements using SDLC best practices.
• Data Quality: Identifies, corrects, and manages data flaws to support effective governance and decision-making.
• Problem Solving: Uses systematic analysis to determine root causes and implement robust solutions.
• Values Differences: Recognizes and leverages the value of diverse perspectives and cultures.