Develop and deploy code in Airflow and Terraform for access management, data controls, and self-service ETL creation
Construct new aggregate tables and join/aggregation logic that is helpful for the analysis and reporting of measurements and marketing
To ingest data into BigQuery through APIs, use Airflow ETL development
Verify that data quality requirements are met (edge cases, sanity check, etc.)
Create Python/SQL monitoring/alerting tools for service health reporting.
Utilize production-related data
Build scalable infrastructure platforms for other engineering and development teams
Interface with third-party vendors
Design and implement tooling that supports dozens of data science, machine learning, and engineering teams across the company
Ingest data from APIs
Translate stakeholder needs into technical solutions by taking part in meetings with stakeholders to discuss technical/operational difficulties and suggest solutions
Job Requirements:
Bachelor s/Master s degree in Engineering, Computer Science (or equivalent experience)
At least 5+ years of relevant experience as a Data Engineer/Analyst
Demonstrable experience working with Python / SQL
Experience managing data infrastructure with cloud services, BigQuery, GCP, Airflow, and Terraform
Design and implement ETL data pipelines
Extensive development experience in pulling data from API and ingesting it into BigQuery
Familiar with data quality concepts (edge cases, sanity check )
In-depth knowledge of Data aggregation as well as Dashboards design and creation
Strong API design, development, and integration experience
Knowledge and experience building user authentication and authorization (IAM setup) between multiple systems, servers, and environments