AWS Data lake Engineer

Job Insight:

We are seeking a skilled and experienced Data Lake Engineer to join our dynamic team.

As a Data Lake Engineer, you will be responsible for designing, implementing, and

maintaining our data lake infrastructure. You will collaborate with data architects, data

scientists, and other stakeholders to ensure efficient data storage, retrieval, and

processing capabilities. The ideal candidate will have at least 1 year of experience in

data engineering with a focus on data lake technologies.

Responsibilities:

1. Design and implement scalable data lake architectures using technologies such as

Glue, EMR and AWS S3.

2. Develop and maintain data ingestion pipelines to efficiently collect and store

structured and unstructured data from various sources.

3. Optimize data lake performance and reliability by implementing best practices in data

partitioning, indexing, and compression.

4. Collaborate with data scientists and analysts to understand data requirements and

implement data transformations and aggregations as needed.

5. Ensure data lake security and compliance with regulatory requirements by

implementing access controls, encryption, and auditing mechanisms.

6. Monitor data lake performance and troubleshoot issues to ensure high availability and

reliability.

7. Document data lake architecture, processes, and procedures for knowledge sharing

and future reference.

8. Stay updated on emerging trends and technologies in data engineering and contribute

to continuous improvement initiatives.

Required Skills:

1. Bachelor'Pys degree in Computer Science, Engineering, or a related field.

2. Minimum 1 year of experience in data engineering with a focus on data lake

technologies.

3. Proficiency in programming languages such as Python, Pyspark, SQL.

4. Hands-on experience with data lake technologies and cloud storage solutions like

AWS S3.

5. Experience with data ETL tools and frameworks such as Apache NiFi, Apache Kafka,

or AWS Glue.

6. Familiarity with data governance, security, and compliance requirements.

7. Excellent problem-solving skills and ability to work effectively in a fast-paced

environment.

8. Strong communication and collaboration skills to work effectively with cross-functional

teams.

Good to have skills:

9. Understanding of data modeling concepts and experience with schema design for

structured and semi-structured data.

10. Experience with Agile process methodology

11. Exposure to Data bricks technology.