Senior Data Engineer with AWS

Key Responsibilities:

• Design, develop, and implement data processing pipelines and workflows using Apache Spark , Scala , and Python to handle large-scale data.
• Work with cloud technologies, primarily AWS , to build and maintain data infrastructures including EC2, S3, Lambda, EMR, and Redshift.
• Manage and integrate MongoDB to store, retrieve, and process large datasets efficiently.
• Collaborate with data scientists, analysts, and other teams to understand data needs and develop solutions that meet business requirements.
• Optimize data processes for scalability and performance by utilizing the best practices for distributed computing.
• Ensure data quality, consistency, and security across all systems.
• Develop automation scripts and monitor system performance to prevent issues and improve the data pipeline process.
• Participate in architecture discussions to design scalable, high-performance data systems.
• Troubleshoot and resolve data-related issues in a timely manner.