Cloud Analogy - Data Engineer - Python (4-6 yrs)

Job Overview :

We are looking for an experienced Data Developer to join our team. The ideal candidate will have a strong background in data development, with expertise in Python and various cloud technologies. This role requires hands-on experience in building data pipelines and designing data management solutions that leverage both relational and non-relational databases. The successful candidate will be a curious technology enthusiast who understands the intricacies of cloud computing, microservices, and streaming technologies.

Key Responsibilities :

1. Data Pipeline Development :

- Design, build, and maintain scalable data pipelines in a cloud environment using tools such as Azure Data Lake Storage, Azure Event Hubs, and Kafka.

- Develop data ingestion processes for structured and unstructured data, ensuring data quality and integrity.

2. Database Management :

- Work with SQL Server, Cosmos DB, and Mongo DB to design and optimize data management solutions.

- Create and maintain database schemas, indexes, and stored procedures to enhance performance and data retrieval.

3. Data Solutions Design :

- Design and develop data persistence solutions that cater to various application use cases, focusing on relational and non-relational databases.

- Enhance data processing capabilities by implementing best practices and leveraging advanced technologies.

4. Streaming Technologies :

- Utilize Spark Streaming and other streaming technologies to handle real-time data processing and event-driven architectures.

- Collaborate with teams to ensure effective integration of streaming solutions with existing systems.

5. Cloud Computing :

- Leverage cloud services, particularly Azure, to support data storage, processing, and analytics needs.

- Stay current with emerging cloud technologies and trends to continuously improve data solutions.

6. Collaboration and Leadership :

- Work closely with cross-functional teams to gather requirements and ensure successful project delivery.

- Act as a hands-on engineering lead, mentoring junior developers and promoting best practices in data development.

Required Skills and Qualifications :

Experience :

- Minimum of 4 years of experience as a data developer, with a strong focus on Python and data pipeline development.

Technical Skills :

- Proficient in Kafka, Spark Streaming, Azure SQL Server, Cosmos DB, and Mongo DB.

- Hands-on experience with Azure services, including Azure Event Hubs and Azure Data Lake Storage.

Data Management :

- Solid understanding of relational and non-relational databases, with experience designing data models and persistence solutions.

- Experience handling unstructured data and working within a data lake environment.

Cloud and Microservices :

- Familiarity with cloud computing concepts, especially Azure (preferred), AWS, and microservices architecture.

- Knowledge of network and security best practices in cloud environments.