Add office photos

Metafolks

Compare

based on 1 Review

142 Metafolks Jobs

Data Engineer

Metafolks

based on 1 Review

1-4 years

Hyderabad / Secunderabad

1 vacancy

Data Engineer

Metafolks

posted 5mon ago

Key skills for the job

+ 3 more

Design and build data pipelines to process terabytes of data
Orchestrate in Airflow the data tasks to run on Kubernetes/Hadoop for the ingestion, processing and cleaning of data.
Create Docker images for various applications and deploy them on Kubernetes
Design and build best in class processes to clean and standardize data.
Troubleshoot production issues in our Elastic Environment
Tuning and optimizing data processes
Advancing the team s DataOps culture (CI/CD, Orchestration, Testing, Monitoring) and building out standard development patterns
Drive innovation by testing new technology and approaches to continually advance the capability of the data engineering function.
Drive efficiencies in current engineering processes via standardization and migration of existing on-premises processes to the cloud
Ensuring Data Quality building best in class data quality monitoring that ensure that all data products exceed customer expectations.

Required Qualifications:

Computer Science bachelor s degree or similar.
Good understanding of Data Modelling techniques i.e. DataVault, Kimble Star
Excellent understanding of Column-Store RDBMS (DataBricks, Snowflake, Redshift, Vertica, Clickhouse)
Good experience handling real-time, near real-time and batch data ingestions
Hands on experience on the following technologies:

Developing processes in Spark
Writing complex SQL queries
Building ETL/data pipelines
Exposure to Kubernetes and Linux containers (i.e. Docker)
Related/complementary open-source software platforms and languages (e.g. Scala, Python, Java, Linux)

Proven track record of designing effective data strategies and leveraging modern data architectures that resulted in business value
Experience building cloud-native data pipelines on either AWS, Azure or GCP, following best practices in cloud deployments
Strong DataOps experience (CI/CD, Orchestration, Testing, Monitoring)
Demonstrated effective interpersonal, influence, collaboration and listening skills
Strong stakeholder management skills
Excellent time management, organizational and prioritization skills with ability to balance multiple priorities.

Preferred Qualifications:

Experience with data tokenization and different techniques and tools i.e. DataVant, Protegrity
Experience with Azure Data Factory, Databricks and Snowflake
Experience with Apache Spark and related Big Data stack and technologies, PySpark Scala
Experience working with Apache Kafka, building appropriate producer/consumer apps
Experience working with Kubernetes and Docker, and knowledgeable about cloud infrastructure automation and management (e.g., Terraform)
Experience working in projects with agile/scrum methodologies
Healthcare industry knowledge and experience with exposure to EDI, HIPAA, HL7 and FHIR integration standards

Employment Type: Full Time, Permanent