Lead Software Engineer - PySpark/AWS (8-13 yrs)

Job Description :

- 8-12 years of good hands on exposure with Big Data technologies - pySpark (Data frame and SparkSQL), Hadoop, and Hive.

- Good hands on experience of python, Bash Scripts.

- Good understanding of SQL and data warehouse concepts

- Strong analytical, problem-solving, data analysis and research skills

- Demonstrable ability to think outside of the box and not be dependent on readily available tools

- Excellent communication, presentation and interpersonal skills are a must

Cloud Skills :

- Hands-on experience with using Cloud Platform provided Big Data technologies (i.e. IAM, Glue, EMR, RedShift, S3, Kinesis)

- Orchestration with Airflow and Any job scheduler experience

- Experience in migrating workload from on-premise to cloud and cloud to cloud migrations

Roles & Responsibilities :

- Develop efficient ETL pipelines as per business requirements, following the development standards and best practices.

- Perform integration testing of different created pipeline in AWS env.

- Provide estimates for development, testing & deployments on different env.

- Participate in code peer reviews to ensure our applications comply with best practices.

- Create cost effective AWS pipeline with required AWS services i.e S3,IAM, Glue, EMR, Redshift etc.

Mandatory Skills - Pyspark, AWS