Data Engineer

Job Description: Data Engineer

Position Overview:

Role Overview

We are seeking a skilled Python Data Engineer with expertise in designing and implementing data solutions using the AWS cloud platform. The ideal candidate will be responsible for building and maintaining scalable, efficient, and secure data pipelines while leveraging Python and AWS services to enable robust data analytics and decision-making processes.

Key Responsibilities

• Design, develop, and optimize data pipelines using Python and AWS services such as Glue, Lambda, S3, EMR, Redshift, Athena, and Kinesis.
• Implement ETL/ELT processes to extract, transform, and load data from various sources into centralized repositories (e.g., data lakes or data warehouses).
• Collaborate with cross-functional teams to understand business requirements and translate them into scalable data solutions.
• Monitor, troubleshoot, and enhance data workflows for performance and cost optimization.
• Ensure data quality and consistency by implementing validation and governance practices.
• Work on data security best practices in compliance with organizational policies and regulations.
• Automate repetitive data engineering tasks using Python scripts and frameworks.
• Leverage CI/CD pipelines for deployment of data workflows on AWS.

Required Skills and Qualifications

• Professional Experience: 5+ years of experience in data engineering or a related field.
• Programming: Strong proficiency in Python, with experience in libraries like pandas, pySpark, or boto3.
• AWS Expertise: Hands-on experience with core AWS services for data engineering, such as:
• AWS Glue for ETL/ELT.
• S3 for storage.
• Redshift or Athena for data warehousing and querying.
• Lambda for serverless compute.
• Kinesis or SNS/SQS for data streaming.
• IAM Roles for security.
• Databases: Proficiency in SQL and experience with relational (e.g., PostgreSQL, MySQL) and NoSQL (e.g., DynamoDB) databases.
• Data Processing: Knowledge of big data frameworks (e.g., Hadoop, Spark) is a plus.
• DevOps: Familiarity with CI/CD pipelines and tools like Jenkins, Git, and CodePipeline.
• Version Control: Proficient with Git-based workflows.
• Problem Solving: Excellent analytical and debugging skills.

Optional Skills

• Knowledge of data modeling and data warehouse design principles.
• Experience with data visualization tools (e.g., Tableau, Power BI).
• Familiarity with containerization (e.g., Docker) and orchestration (e.g., Kubernetes).
• Exposure to other programming languages like Scala or Java.

Education

• Bachelors or Master’s degree in Computer Science, Engineering, or a related field.

Why Join Us?

• Opportunity to work on cutting-edge AWS technologies.
• Collaborative and innovative work environment.