Senior Data Engineer

• Among other things, use Python, SQL, AWS Glue, lambdas, API Gateway, Kafka, data build tool (dbt), and Apache Spark on EMR to create a modular collection of data services
• Utilizing frameworks such as PySpark, create automated pipelines for unit and integration testing
• Using Gitlab CI and AWS Code Pipeline/CodeDeploy, create and manage CI/CD pipelines
• Schedule and automate tasks with Managed Airflow
• Create the ODS and reporting schemas, then import the data into Snowflake or AWS Redshift
• Using Apache Deequ and data observability technologies such as Splunk, DataDog, CloudWatch, and Montecarlo, design and implement data quality management services
• Offer a range of query services using server-sent events, Athena/Presto, and REST
• Enable numerous technical, operational, and business teams at the client to share data in a fast, efficient, and secure manner in order to create world-class data products
• Utilizing tools such as Collibra/Alation, configure and set up enterprise data lineage, metadata management, and data catalog support
• Make sure deployment builds are green, and collaborate with senior leads and program management to exhaust scheduled deliverables within a sprint cycle

Job Requirements:

• Bachelor s/Master s degree in Engineering, Computer Science (or equivalent experience)
• At least 4+ years of relevant experience as a data engineer
• Prolific understanding of data engineering, AWS, and PySpark
• Some prior experience with Apache Kafka, SQL, AWS Lambda, and CI/CD is desirable
• Great collaboration and team-building skills
• Excellent English communication skills, both written and conversational