Labcorp - Senior Data Analytics Engineer - Big Data/PySpark (5-7 yrs)

Job Responsibilities :

- Designing and implementing highly performant data ingestion pipelines from multiple sources

- Delivering and presenting proofs of concept to of key technology components to project stakeholders.

- Developing scalable and re-usable frameworks for ingesting of data sets

- Integrating the end-to-end data pipeline to take data from source systems to target data repositories ensuring the quality and consistency of data is maintained at all times

- Working with event based / streaming technologies to ingest and process data

- Do analysis of new requirements and propose a solution

- Evaluating the performance and applicability of multiple tools against customer requirements

Required Skills :

- At least 4 years of Hands-on experience on PySpark, Python, Spark, Scala, Able to perform data validation, delivery, quality and integrity.

- At least 4 years of Snowflake /Databricks Cloud Data warehouse Implementation experience, Should possess skill pertaining to handling various file formats (XML,JSON,Avro,Parquet)

- Good Experience in Extraction, Transformation and Loading (ETL) data from multiple data sources & various data formats (Structured, Semi-Structured & Unstructured)

- Should be able to work with modern schedulers (Airflow, Matillion, etc.) and distributed data systems such as Hadoop, Spark, Kafka etc.

- Should have experience in business intelligence, reporting, and/or delivery of modern visualization solutions (Tableau, Power BI)

- Good working exposure on cloud technologies of AWS - EC2, S3, Lambda, SQS, SNS

- Strong Programming experience using PL/SQL packages, Stored procedures, views, Materialized views, triggers, Indexes etc.

- Strong understanding on Data Warehousing concepts using Fact, Dimension tables and various methods of loading data

- Hands-on Experience with Big Data model techniques using Python / Spark

- Experience in CI/CD domain using Jenkins, Terraform, Docker, AWS, and Kubernetes.

- Experience in Pharma industry is a plus

- Strong communication and collaboration skills.