Infogain - Data Engineer - PySpark/Python (6-11 yrs)

Core Skills :

Exp : 5-10 years Must have - Python, Java, Bigquery, PySpark

- Extensive experience with Google Cloud Platform (GCP) data services such as BigQuery, Cloud Storage, and Dataflow.

- Expertise in ETL (Extract, Transform, Load) processes and data integration on GCP.

- Strong SQL and database querying optimization skills on GCP.

- Experience with data warehousing and data architecture on GCP.

Responsibilities :

- Design, code, and develop new features/fix bugs/add enhancements.

- Analyze and improve the efficiency, scalability and stability of various system resources.

- Lead and Mentor junior engineers and drive a culture of technical perfection.

- Drive creative and innovative solutions to complex problems, exemplifying good technical discernment.

- Drive improvements and new approaches to address potential systemic pain points and technical debt, anticipate and avoid problems.

- Take a hands-on approach in developing prototypes, independently and with others, to establish design decisions and/or technical feasibility.

- Evaluate, Install, Setup, Maintain and Upgrade, Data Engineering, Machine Learning and CI/CD infrastructure tools hosted on Cloud (GCP/AWS)

- Drive the CI/CD infrastructure tooling-related work in collaboration with various internal teams to get the user stories, epics, and goals to closure.

- Propose, participate, and implement architecture-level enhancements/changes strategically through Dev/Stage/Prod environments.

- Design, evangelize, deliver comprehensive best practices & efficient usage of available tooling resources/capabilities to run high-performance systems.

- Provide innovative & and strategic solutions, along with cost & risk analysis, to improve the stability, scalability, and performance of the tools" infrastructure.

- Perform troubleshooting, analysis, and resolution of environmental issues.

- Innovate and automate processes/tasks to improve operational efficiency

- Document and maintain application setup, runbooks, administration, and troubleshooting guides.

Programming Skills :

- Algorithms

- Design Skills

- Refactoring Skills

- Debugging Skills

- Unit testing

Skills :

- Database

- SQL Databases (Oracle/MySQL/MSSQL etc.,)

- No SQL Databases

- Data Lake, Snowflake