Perfios Software Solutions - Tech Lead - Data Engineering (6-8 yrs)

Job Description : Tech Lead Data Engineer

Position Overview :

We are looking for a seasoned Tech Lead Data Engineer with a robust technical background and leadership experience to spearhead our data engineering efforts.

This role demands a deep expertise in Python, PySpark, and Shell scripting, along with a comprehensive understanding of AWS cloud services. As a Tech Lead, you will manage a team of data engineers, oversee the development and maintenance of scalable data solutions, and ensure the seamless integration and deployment of projects.

You will play a key role in optimizing our AWS infrastructure, implementing CI/CD pipelines, and managing APIs and serverless architectures. This position requires not only strong technical skills but also the ability to lead projects, mentor team members, and drive innovation within the data engineering domain.

Key Responsibilities :

Team Leadership :

- Lead, mentor, and manage a team of data engineers, providing guidance on technical and project-related matters.

- Foster a collaborative team environment, ensuring effective communication and knowledge sharing.

Data Engineering & Development :

- Design, develop, and optimize data pipelines and workflows using Python, PySpark, and Shell scripting.

- Leverage key data engineering libraries in Python, such as Pandas, Dask, and Polars, to build and maintain scalable data solutions.

- Apply best practices for writing clean, efficient, and optimized Python code.

AWS Infrastructure Management :

- Architect, implement, and maintain infrastructure on AWS, including services such as S3, Lambda, CloudWatch, SQS, Kinesis, EventBridge, EC2, EMR, AWS Glue, Athena, CloudFormation, and Secrets Manager.

- Manage and monitor APIs running on AWS API Gateway, ensuring high availability and performance.

- Implement logging, debugging, and monitoring practices to maintain code quality and system reliability.

- Track and optimize metrics and costs associated with AWS services.

Project & Deployment Management :

- Oversee project management on GitLab, including version control, branch management, and code reviews.

- Handle deployments using serverless technologies and manage CI/CD pipelines, ensuring smooth and efficient releases.

- Utilize Docker containers to build, test, and deploy applications in a scalable and consistent environment.

Database Management :

- Work with SQL and NoSQL databases (MongoDB) to design and manage efficient data storage and retrieval systems.

- Ensure data integrity, performance optimization, and security across all database systems.

Regular Expressions (Regex) :

- Utilize strong understanding and proficiency in regex to extract, manipulate, and validate data efficiently within various applications and pipelines.

Required Skills and Qualifications :

Technical Proficiency :

- Strong experience in Python and PySpark for data processing and analysis.

- Proficiency in Shell scripting for automation and task management.

- Expertise in SQL and NoSQL databases, particularly MongoDB.

- Strong understanding and experience with regular expressions (regex) for data extraction and manipulation.

- Familiarity with key Python libraries for data engineering, such as Pandas, Dask, and Polars.

- Experience in optimizing Python scripts using best practices.

AWS Expertise :

- In-depth knowledge of AWS services, including S3, Lambda, CloudWatch, SQS, Kinesis, EventBridge, EC2, EMR, AWS Glue, Athena, CloudFormation, and Secrets Manager.

- Experience in monitoring, logging, debugging, and maintaining AWS infrastructure.

- Ability to track and optimize AWS metrics and costs effectively.

Project Management :

- Experience managing projects on GitLab, including code versioning, branch management, and collaboration.

- Knowledge of CI/CD pipelines and serverless deployment strategies.

- Proficiency in using Docker containers for application development and deployment.

Leadership & Communication :

- Proven ability to lead and mentor a small team of engineers.

- Strong communication skills, with the ability to articulate technical concepts to non-technical stakeholders.

- Experience in managing and delivering multiple projects simultaneously.

Basic Machine Learning Knowledge (Preferred) :

- Understanding of machine learning fundamentals and how they apply to data engineering tasks.

Preferred Qualifications :

- AWS Certified Solutions Architect or Data Engineer certification.

- Experience with additional AWS services beyond the core stack mentioned.

- Familiarity with other big data technologies and frameworks.

Why Join Us?

- Work with cutting-edge technologies in a collaborative and innovative environment.

- Lead a talented team of data engineers and contribute to impactful projects.

- Competitive salary and benefits package, with opportunities for career growth.

If you are a highly motivated and experienced Tech Lead Data Engineer with a passion for data and cloud technologies, we would love to hear from you. Apply today and help us shape the future of our data infrastructure!