Sr. Cloud Data Engineer (AWS-Big Data)

Position: Sr. Cloud Data Engineer- AWS

Location: Nagpur/Pune/Chennai/Bangalore

Type of Employment: Full-time

Purpose of the Position:

As a Sr. Cloud Data Engineer, this position requires candidate who are enthusiastic about data and driven to help in organization's AI & Analytics challenges. You will help our clients on their data and analytics journey, by building data pipelines, improving consistency, and building the necessary infrastructure to support their analytics platform. 

Key Result Areas and Activities:

Share and Build Expertise- Develop and share expertise in AWS and Python and actively mine the experience and expertise in the organization for sharing across teams and clients in the firm. Support the Cloud and Data Engineering COE initiatives.

Assessment of Technology Landscape and Data Integration Framework-

• Impact Analysis: Evaluate the current technology stack and data integration frameworks to assess the impact of new project requirements on existing systems.
• Framework Evaluation: Identify and address gaps or opportunities for improvement in the existing data integration technologies and processes.

Development and Optimization of Spark Jobs-

• New Spark Job Development: Design and implement Spark jobs using Scala to meet emerging project requirements.
• Enhancement of Existing Jobs: Refine and optimize current Spark jobs to support ongoing product enhancements and ensure they meet evolving needs.
• Performance Tuning: Execute performance tuning, stress testing, and optimizations to enhance the efficiency and effectiveness of Spark jobs.

Creation of Data Pipelines and Database Design-

• Data Pipeline Development: Develop new data pipelines for Spark jobs using AWS services such as AWS Lambda or orchestrate with Apache Airflow.
• Database Design: Lead the database design process, including the creation of logical and physical designs, and schema types like star and snowflake schemas.
• Data Loading Strategies: Define and implement strategies for data loading that consider volume, data types, frequency, and analytics needs for optimal data processing and integration.

Essential Skills:

• Strong experience in Python and Spark  
• Strong understanding of cloud and data engineering concepts in AWS 
• Implement and manage data processing pipelines using Python libraries and AWS services such as AWS Lambda, Amazon S3, SNS, SQS and Athena 
• Implement and manage serverless applications using AWS Lambda and other serverless computing services 
• Implement and manage AWS storage services, including Amazon S3, Amazon EBS, and Amazon RDS 
• Good expertise on DWH such as Snowflake, Redshift, Bigquery, etc.  
• Use version control systems such as Git to manage codebase, track changes, and collaborate effectively with other team members
• Expertise in developing ETL workflows comprising complex transformations like SCD, deduplications, aggregations, etc. 
• Develop and maintain scalable data pipelines and builds out new API integrations to support continuing increases in data volume and complexity

Desirable Skills:

• Experience and technical knowledge in Databricks 
• Strong Experience with event stream processing technologies such as Kafka, KDS 
• Knowledge of Operating System- Any flavor of Linux, ETL Tools Informatica, Talend 

Qualifications:

• Bachelors degree in computer science, engineering, or related field (Masters degree is a plus) 
• 7 to 10 years of IT experience including 3+ years of AWS cloud experience 
• Experience with or knowledge of Agile Software Development methodologies 

Qualities:

• Self-motivated and focused on delivering outcomes for a fast growing team and firm
• Able to communicate persuasively through speaking, writing, and client presentations
• Able to consult, write, and present persuasively
• Excellent verbal and written communication skills