Work with diverse functional teams to coordinate and deliver cohesive data engineering solutions across many projects and endeavors
Hands-on experience working on structured/semi-structured
Help the team to execute the Data Engineering and Analytics initiatives and turn them into products that will provide great value to our members
Complex data processing and streaming frameworks
Work with other technical teams to produce data solutions that satisfy business and technical objectives by being hands-on and taking ownership of the whole cycle of data services, from data ingestion to data processing to ETL to data distribution for reporting
Establish the technical specifications and specifics of the implementation for the underlying data lake, data warehouse, and data marts
Perform data source gap analysis and create data source/target to identify, troubleshoot, and fix production data integrity and performance issues
Collaborate with all areas of data management as lead to ensure that patterns, decisions, and tooling are implemented in accordance with enterprise standards
To establish an enterprise picture of the company's future data demands, thoroughly grasp cross-system integration, interactions, and relationships
Recommend/Ensure technical functionality (e.g. scalability, security, performance, data recovery, reliability, etc.) for Data Engineering
Facilitate workshops to define requirements and develop data solution designs
Apply enterprise and solution architecture decisions to data architecture frameworks and data models
Maintain a repository efficiently
Design, coordinate, and execute pilots, prototypes, and proofs of concept to provide validation on specific scenarios and provide an implementation roadmap
Work together to predict and create data architecture that meets business demands for the collection, aggregation, and interaction with multiple data sources with IT teams software suppliers and business owners
Job Requirements:
Bachelor s/Master s degree in Engineering, Computer Science (or equivalent experience)
At least 5+ years of relevant experience as a data engineer
Extensive experience working with Python, Scala, Airflow, and SQL
Prolific experience with Data warehousing/modeling
Nice to have some experience with Git, AWS Redshift, and Microservices
Familiarity with DBT (Data Build Tool) is desirable
Ability to work cross-functionally with other engineering teams
Expertise with RDBMS and Data Warehousing (Strong SQL) with Redshift, Snowflake, or similar
In-depth knowledge and experience with data and information architecture patterns and implementation approaches for Operational Data Stores, Data Warehouses, Data Marts, and Data Lakes
Proficiency in logical/physical data architecture, design, and development
Experience in Data lake / Big data analytics platform implementation either cloud-based or on-premise; AWS preferred
Experience working with high volumes of data; experience in design, implementation, and support of highly distributed data applications
Experience with Development Tools for CI/CD, Unit and Integration testing,
Automation and Orchestration E.g. GitHub, Jenkins, Concourse, Airflow, Terraform
Experience with writing Kafka producers and consumers or experience with AWS Kinesis
Hands-on experience developing a distributed data processing platform with Big Data technologies like Hadoop, Spark, etc
A knack for independence (hands-on) as well as team work
Excellent analytical and problem-solving skills, often in light of ill-defined issues or conflicting information.
Experience with streaming data ingestion, machine learning, and Apache Spark a plus
Adept in the ability to elicit, gather, and manage requirements in an Agile delivery environment
Excellent communication and presentation skills (verbal, written, presentation) across all levels of the organization
Ability to translate ambiguous concepts into tangible ideas