Data Engineer

GPP Database Link

Job Summary:

Supports, develops, and maintains a data and analytics platform to effectively and efficiently process, store, and make data available to analysts and other consumers. Works with Business and IT teams to understand requirements and best leverage technologies for agile data delivery at scale. Though the role category is listed as Remote, this specific position is designated as Hybrid.

Key Responsibilities:

• Implement and automate deployment of distributed systems for ingesting and transforming data from various sources (relational, event-based, unstructured).
• Develop and maintain data pipelines using ETL/ELT tools or scripting languages, ensuring reliability, efficiency, scalability, and quality with monitoring and alert mechanisms.
• Monitor and troubleshoot data quality and integrity issues.
• Implement data governance processes, including metadata management, access control, and retention policies for internal and external users.
• Develop physical data models and implement data storage architectures following design guidelines.
• Analyze complex data elements, systems, flows, dependencies, and relationships to contribute to conceptual, physical, and logical data models.
• Participate in testing and troubleshooting of data pipelines.
• Operate large-scale data storage and processing solutions using distributed and cloud-based platforms (e. g. , Data Lakes, Hadoop, HBase, Cassandra, MongoDB, Accumulo, DynamoDB).
• Utilize agile development technologies, including DevOps, Scrum, and Kanban, to drive continuous improvement in data-driven applications.

Preferred Experience:

• Relevant experience through internships, co-ops, temporary student employment, or extracurricular team activities.
• Exposure to Big Data open-source technologies such as Spark, Scala/Java, Map-Reduce, Hive, HBase, and Kafka.
• Hands-on experience with SQL query language.
• Experience in clustered compute cloud-based implementation.
• Familiarity with developing applications requiring large file movement in a cloud-based environment.
• Exposure to Agile software development methodologies.
• Experience in building analytical solutions.
• Exposure to IoT technology.

Technical Skills:

1. Programming Languages: Proficiency in Python, Java, and/or Scala.
2. Database Management: Expertise in SQL and NoSQL databases.
3. Big Data Technologies: Hands-on experience with Hadoop, Spark, Kafka, and similar frameworks.
4. Cloud Services: Experience with Azure, Databricks, and AWS platforms.
5. ETL Processes: Strong understanding of Extract, Transform, Load (ETL) processes.
6. Data Replication: Working knowledge of replication technologies like Qlik Replicate is a plus.
7. API Integration: Experience working with APIs to consume data from ERP and CRM systems.

Core Competencies:

• System Requirements Engineering: Translate stakeholder needs into verifiable requirements, establish acceptance criteria, track requirement status, and assess impact changes.
• Collaborates: Build partnerships and work collaboratively with others to meet shared objectives.
• Communicates Effectively: Deliver multi-mode communications tailored to different audiences.
• Customer Focus: Build strong customer relationships and provide customer-centric solutions.
• Decision Quality: Make good and timely decisions that drive the organization forward.
• Data Extraction: Perform ETL activities from various sources using appropriate tools and technologies.
• Programming: Develop, test, and maintain code using industry standards, version control, and automation tools.
• Quality Assurance Metrics: Measure and assess solution effectiveness using IT Operating Model (ITOM) standards.
• Solution Documentation: Document knowledge gained and communicate solutions for improved productivity.
• Solution Validation Testing: Validate configurations and solutions to meet customer requirements using SDLC best practices.
• Data Quality: Identify, correct, and manage data flaws to support effective governance and decision-making.
• Problem Solving: Use systematic analysis to determine root causes and implement robust solutions.
• Values Differences: Recognize and leverage the value of diverse perspectives and cultures.

Education, Licenses, and Certifications:

• Bachelors degree in a relevant technical discipline, or equivalent experience required.
• This position may require licensing for compliance with export controls or sanctions regulations.