Innovatily - Data Engineer - ETL Pipeline (3-6 yrs)

About Us :

Innovatily is dedicated to transforming data into actionable insights. We leverage cutting-edge technologies like Apache Spark and AWS Glue to build robust, scalable ETL pipelines that support our diverse data processing needs. We are seeking talented ETL Developers to join our team and help us scale our data infrastructure.

Job Description :

As an ETL Developer, you will be responsible for designing, building, and maintaining ETL pipelines using Apache Spark and AWS Glue. You will work closely with data engineers, analysts, and other stakeholders to ensure data is accessible, reliable, and processed efficiently.

Key Responsibilities :

- Design, develop, and deploy ETL pipelines using Apache Spark and AWS Glue.

- Collaborate with data engineers and analysts to understand data requirements and translate them into efficient ETL processes.

- Optimize data pipelines for performance, scalability, and robustness.

- Integrate data from various sources, including S3, relational databases, and APIs.

- Implement data validation and error handling mechanisms to ensure data quality.

- Monitor and troubleshoot ETL jobs to ensure data accuracy and pipeline reliability.

- Maintain and update existing data pipelines as data sources and requirements evolve.

- Document ETL processes, data models, and pipeline configurations.

Qualifications :

- Experience: 3+ years of experience in ETL development, with a focus on Apache Spark and AWS Glue.

Technical Skills :

- Strong expertise in Apache Spark (PySpark, Spark SQL) and AWS Glue.

- Proficiency in programming languages such as Python or Scala.

- Hands-on experience with AWS services like S3, Lambda, and IAM.

- Solid understanding of data warehousing concepts, data modeling, and ETL best practices.

- Familiarity with orchestration tools like Apache Airflow is a plus.

- Data Integration: Experience with integrating data from diverse sources, including relational databases, APIs, and flat files.

- Problem-Solving: Strong analytical and problem-solving skills with the ability to troubleshoot complex ETL issues.

- Communication: Excellent communication skills, with the ability to work collaboratively with cross-functional teams.

- Education: Bachelor's degree in computer science, Engineering, or a related field, or equivalent work experience.

Nice to Have :

- Experience with data lakes and big data processing.

- Familiarity with Glue Data Catalog, Glue Studio, and Glue Workflows.

- Knowledge of data governance and security practices in a cloud environment.