Data Engineer/Cloud Engineer

As a Cloud Developer specializing in data practices, you will be pivotal in developing, migrating and optimizing data-intensive applications and solutions on cloud platforms. Leveraging your expertise in cloud technologies and data engineering principles, you will implement efficient solutions. Your innovative mindset and proficiency in cloud platforms will drive efficiency and agility within our development teams.

Responsibilities:

• Collaborate as part of a development team to design and enhance large scale applications developed using Python, Spark.
• Evaluates and plans software designs, test results and technical manuals using AWS
• Confer with business units and development staff to understand both the business and technical requirements for producing technical solutions.
• Create and review technical and user-focused documentation for data solutions (data models, data dictionaries, business glossaries, process and data flows, architecture diagrams, etc.).
• Extend and enhance the business Data Lake
• Create or implement solutions for metadata management
• Solve for complex data integrations across multiple systems.
• Design and execute strategies for real-time data analysis and decisioning.
• Build robust data processing pipelines using AWS Services and integrate with multiple data sources
• Translating client user requirements into data flows, data mapping, etc.
• Analyses and determines data integration needs and follows Agile practices.

Required Skills:

• Experience on designing and developing Data Pipelines for Data Ingestion or Transformation using Scala or Python
• Experience with Python, Spark.
• At least 3 years of experience working on AWS technologies.
• Experience of designing, building, and deploying production-level data pipelines using tools from AWS Glue, Lamda, Kinesis using databases Aurora and Redshift.
• Experience with Spark programming (pyspark or scala).
• Hands on experience with AWS components like (EMR, S3, Redshift, Lamdba, API Gateway, Kinesis ) in production environments
• Strong analytical skills and advanced SQL knowledge, indexing, query optimization techniques.
• Experience using ETL tools for data ingestion..
• Experience with Change Data Capture (CDC) technologies and relational databases such as MS SQL, Oracle and DB
• Ability to translate data needs into detailed functional and technical designs for development, testing and implementation
• 5+ Years of Development / Data Engineering Experience
• 4+ Years Developing with AWS Cloud solutions
• 4+ Years Developing in Spark/PySpark
• 4+ Years Developing AWS Glue ETL
• 4 + Years AWS Storage models (S3 and DynamoDB)
• Some hands-on developing with on-prem ETL tooling (Ab Initio preferred, Informatica)

Additional Skills:

• Strong experience in AWS GLUE, EMR, Hudi, experience extracting data from multiple sources & loading data into Data Lakes, AWS Redshift
• Experience working with AWS Elastic search, RDS, PostgreSQL preferred
• Experience w/ AWS services such as Lambda, EMR, SNS/SQS, Event Bridge, Lake Formation & Athena
• Experience w/ integrating applications/systems (data producers) with Enterprise Kafka topics (Confluent Kafka integration w/ AWS S3, Redshift)
• Experience in Requirements Analysis, Data Analysis, Application Design, Application Development & Integration Testing
• Working knowledge of on-prem Extraction, Transformation, Cleansing and Loading methodology and principles
• Experience implementing and adding to DevOps principles (GitLab, maintaining CI/CD pipelines)