AWS- Bigdata Engineer

Location: Nagpur/Pune/Chennai/Bangalore

Type of Employment: Full-time

Key Result Areas and Activities:

• Study existing technology landscape and understand current data integration framework and
  do impact assessment for the requirements.
• Develop spark jobs using Scala for new project requirements.
• Enhance existing spark jobs for any ongoing product enhancement.
• Performance tuning of spark jobs, stress testing etc.
• Create new data pipelines for developed/enhanced spark jobs using AWS Lambda or Apache Airflow
• Responsible for database design process, logical design, physical design, star schema, snowflake schema etc.
• Analyze data processing, integration, modelling and reporting requirements & Define data loading strategies considering volume, data types, frequency and analytics specifications.
• Ensure optimal balance between cost and performance.
• Project documentation, Adheres to Quality guidelines & Schedules.
• Works hand in hand with PM for successful delivery of project and provide Estimation, scoping, scheduling assistance.
• Manage build phase and quality assure code to ensure fulfilling requirements and adhering to Cloud architecture. Resolve difficult design and develop issues.

Work and Technical Experience:

Essential Skills:

• Knowledge of following AWS Services required: S3, EC2, EMR, Severless, Athena, AWS Glue,
  Lambda, Step Functions
• Cloud Databases AWS Aurora, Singlestore, RedShift, Snowflake
• Big Data- Hadoop, Hive, Spark, YARN
• Programming Language Scala, Python, Shell Scripts, PySpark
• Operating System- Any flavor of Linux, Windows
• Strong SQL Skills
• Orchestration Tools: Apache Airflow
• Hands-on in developing ETL workflows comprising complex transformations like SCD, deduplications, aggregations etc.

Qualifications:

• Bachelors degree in computer science, engineering, or related field (Masters degree is a plus)
• Demonstrated continued learning through one or more technical certifications or related methods
• At least 5+ years of IT experience; Minimum 3 years of experience on Cloud related projects

Qualities:

• Hold strong technical knowledge and experience
• Should have the capability to deep dive and research in various technical related fields
• Self-motivated and focused on delivering outcomes for a fast growing team and firm
• Able to communicate persuasively through speaking, writing, and client presentations
• Able to consult, write, and present persuasively
• Able to work in a self-organized and cross-functional team
• Able to iterate based on new information, peer reviews, and feedback
• Prior experience of working in a large media company would be added advantage