DevOps/DataOps Developer - Python/Spark (5-10 yrs)

Location : Hybrid. Maybe once a week from Office (Pune / Mumbai preferred)

(New York Time Zone 12noon - 9pm or 1- 10 pm IST)

Strong Foundation -

- Data Transformation using Python, Spark, Glue, Lambda

- Terraform

- Data Lake formation

- Pipeline creation

- Devops experience in creation of data workstreams

Duties to be Performed :

1. Data transformations using python, spark, Glue, Lambda, etc.

2. Terraform creation for AWS Data services: Glue, Lambda, LakeFormation, etc.

3. Pipeline creation to run the terraform deploys for the above terraform code.

4. Collaborate with AWS and Team leads to identify the infrastructure and functional requirements for the Data workstream

5. Collaborate with Data and Analytics engineers on required AWS resources needed for target infrastructure

6. Collaborate with DevOps engineers in the creation of the Data workstream environment

7. Provide support to overall workstream where required

Required Skills and Experiences :

- Understanding of deploying IaC with Terraform

- Experience with deploying/utilizing the following AWS services :

1. EMR / Glue / Spark

2. Data Lake / Lake Formation

3. IAM , KMS

4. Lambda

5. SQS

6. Athena

7. Kinesis Firehose or Kafka

8. S3

9. Eventbridge

10. Aurora Postgres

11. DynamoDB

12. Redis

13. Snowflake

14. AWS Data Services

- Experience with complex Data Transformations using ETL services such as Glue and Spark

- Experience with Data Definition/Description Language (DDL)

- Strong Python experience for Data manipulation/ETL

- Terraform

- Understanding of AWS network security

Technical Environment :

- Data Lake Environment with the following AWS services :

1. EMR / Glue / Spark o Data Lake / Lake Formation

2. Lambda

3. Athena

4. Kinesis Firehose or Kafka

5. S3 o

6. Aurora Postgres

7. DynamoDB

8. Redis

9. Snowflake