Data Engineer - ETL (4-6 yrs)

Job Title : Data Engineer

Location : Bangalore

Experience Required : 4-6 years

Key Responsibilities :

1. Data Pipeline Development :

- Design, develop, and manage scalable ETL (Extract, Transform, Load) processes to handle large volumes of data efficiently.

- Build and maintain batch and real-time data pipelines using Python and PySpark.

- Leverage Azure Databricks to implement data transformation and integration workflows.

2. Data Integration :

- Integrate data from various sources, including relational databases, APIs, flat files, and cloud-based systems.

- Develop data models and schemas optimized for analytical and transactional workloads.

- Implement and manage Azure Data Factory for orchestrating data workflows.

3. Data Storage and Management :

- Design and implement scalable data storage solutions using Azure Data Lake, Azure SQL Database, and other Azure data services.

- Ensure data quality and integrity through validation checks and error-handling mechanisms.

- Develop partitioning and indexing strategies for efficient data retrieval and storage.

4. Performance Optimization :

- Optimize PySpark code and data pipelines for performance and cost-efficiency.

- Monitor and troubleshoot performance bottlenecks in data workflows.

- Implement caching strategies and parallel processing techniques to enhance throughput.

5. Security and Compliance :

- Implement data security best practices, including encryption, masking, and access control.

- Ensure compliance with data privacy regulations like GDPR, CCPA, etc.

- Develop data governance policies for metadata management and data lineage.

6. Collaboration and Support :

- Work closely with data scientists and analysts to translate business requirements into technical solutions.

- Provide support for analytical and machine learning models by delivering high-quality, processed data.

- Document data processes, architecture, and pipelines for technical and non-technical stakeholders.

Key Skills and Competencies :

Technical Skills :

- Programming : Proficiency in Python with experience in libraries such as Pandas, NumPy, and PySpark.

- Big Data Technologies : Hands-on experience with Apache Spark and Azure Databricks.

- Cloud Platforms : Proficiency in Azure cloud services, including Azure Data Factory, Azure Data Lake, and Azure Synapse.

- Database Management : Strong understanding of SQL and experience with relational (e.g., MySQL, PostgreSQL) and NoSQL databases (e.g., MongoDB, Cassandra).

- ETL Tools : Experience with data integration tools like Informatica, Talend, or native Azure tools.

- Automation : Knowledge of CI/CD practices and tools like Jenkins, Git, and Azure DevOps.

Analytical and Problem-Solving Skills :

- Ability to diagnose and resolve data processing and performance issues.

- Experience in data modeling and analysis for complex datasets.

Communication Skills :

- Strong verbal and written communication skills.

- Ability to present technical concepts to non-technical audiences.

Soft Skills :

- Proactive and detail-oriented with a passion for data-driven decision-making.

- Ability to work independently as well as collaboratively in a fast-paced environment.

Preferred Qualifications :

- Experience with other cloud platforms such as AWS or Google Cloud.

- Familiarity with containerization tools like Docker and orchestration tools like Kubernetes.

- Knowledge of machine learning and AI frameworks is a plus.

- Certifications in Azure data services (e.g., Microsoft Certified: Azure Data Engineer Associate).

Benefits :

- Competitive salary and performance-based bonuses.

- Comprehensive health, dental, and vision insurance.

- Opportunities for professional development and certifications.

- Flexible work hours and remote work options.

- Employee wellness programs and team-building activities.