Kipi.ai - Data Engineer - ETL (4-5 yrs)

About the Role :

We are seeking a highly skilled and motivated Data Engineer to join our growing data team.

As a Data Engineer, you will play a critical role in designing, developing, and maintaining our data infrastructure.

You will be responsible for building and managing data pipelines, transforming data, and ensuring the availability of high-quality data for analysis and decision-making.

Key Responsibilities :

- Design, develop, and maintain data pipelines using a variety of tools and technologies (e.g, Apache Spark, Kafka, Airflow).

- Extract, transform, and load (ETL) data from various sources, including databases, APIs, and cloud storage.

- Build and maintain data warehouses and data lakes.

- Develop and implement data quality checks and data validation procedures.

- Design and implement data warehousing and data lake architectures.

- Optimize data storage and retrieval for efficient data analysis and query performance.

- Implement data security and access controls to ensure data privacy and compliance.

- Experience with cloud platforms (AWS, Azure, GCP) and their data services (e.g, AWS S3, Azure Data Lake Storage, Google Cloud Storage).

- Leverage cloud-native technologies to build scalable and cost-effective data solutions.

- Collaborate with data analysts, data scientists, and business stakeholders to understand data requirements and translate them into technical solutions.

- Communicate technical information clearly and concisely to both technical and non-technical audiences.

- Participate in code reviews and contribute to a collaborative team environment.

- Stay up-to-date with the latest advancements in data engineering technologies and best practices.

- Explore and evaluate new technologies and tools to improve data processing and analysis capabilities.

- Contribute to the development and improvement of data engineering processes and best practices within the organization.

Qualifications :

Essential :

- Bachelor's degree in Computer Science, Engineering, or a related field.

- 4+ years of experience as a Data Engineer or in a related data engineering role.

- Strong proficiency in one or more programming languages (e.g, Python, Java, Scala).

- Experience with data processing frameworks such as Apache Spark, Apache Kafka, or similar.

- Experience with SQL and NoSQL databases (e.g, MySQL, PostgreSQL, MongoDB).

- Experience with data warehousing and data lake concepts and technologies.

- Strong analytical and problem-solving skills.

- Excellent communication and interpersonal skills.

Preferred :

- Experience with cloud platforms (AWS, Azure, GCP) and their data services.

- Experience with data visualization and business intelligence tools (e.g, Tableau, Power BI).

- Experience with data streaming technologies.

- Experience with containerization and orchestration technologies (e.g, Docker, Kubernetes).

- Experience with data security and privacy best practices