Accolite - Data Engineer - SQL/Python (3-8 yrs)

JOB TITLE : Data Engineer

EXPERIENCE : 4-7 years

LOCATION : Bangalore

The Data Engineer will design, develop, and maintain scalable data pipelines and ETL processes using tools like Airflow and PySpark. The ideal candidate will have experience with batch processing and be proficient in AWS. You will collaborate with data scientists and analysts to ensure data availability and integrity for various applications.

Key Responsibilities :

Data Pipeline Development :

- Design and implement robust data pipelines to extract, transform, and load (ETL) data from various sources.

- Utilize Airflow for orchestrating complex workflows and managing data pipelines.

Data Processing :

- Implement batch processing techniques using PySpark to handle large volumes of data efficiently.

- Develop ELT processes to optimize data transformation within the target data warehouse.

AWS Integration :

- Leverage AWS services (e.g., S3, Redshift, Lambda) for data storage, processing, and orchestration.

- Ensure data security, reliability, and performance when utilizing AWS resources.

Collaboration :

- Work closely with data scientists, analysts, and other stakeholders to understand data requirements and provide the necessary data infrastructure.

- Assist in troubleshooting and optimizing existing data workflows

Documentation & Best Practices :

- Create and maintain documentation for data pipelines, workflows, and processes.

- Adhere to best practices in data engineering, including version control and monitoring.

Technical Skills :

- Strong proficiency in building data pipelines using Airflow.

- Experience with ETL and ELT concepts and tools.

- Proficient in batch processing frameworks, particularly PySpark.

- Extensive knowledge of AWS services for data processing and storage.

- Familiarity with SQL and data warehousing concepts.