HiLabs - Data Engineer - Spark/Python (4-6 yrs)

Job Description :

Key Responsibilities :

Data Pipeline Development :

- Design, develop, and optimize scalable ETL (Extract, Transform, Load) processes to ingest and process data from diverse sources.

- Implement real-time and batch data processing systems using tools like Apache Spark, Kafka, or similar technologies.

Data Warehousing :

- Build and maintain data warehouses/lakes for structured and unstructured data, leveraging platforms such as AWS Redshift, Snowflake, or Google BigQuery.

- Design and implement database schemas to ensure efficient querying and data retrieval.

Data Quality and Governance :

- Ensure data quality by implementing monitoring, validation, and auditing frameworks.

- Collaborate on data governance strategies to maintain compliance with organizational and regulatory requirements.

Performance Optimization :

- Optimize database performance and query execution for large-scale datasets.

- Implement caching and indexing strategies to enhance system responsiveness.

Tools and Technology :

- Develop solutions using programming languages such as Python, Scala, or Java.

- Leverage cloud platforms (AWS, Azure, or GCP) for scalable and secure data storage and processing.

Required Qualifications :

Education : Bachelor's or Master's degree in Computer Science, Engineering, or a related field.

Experience :

- 4-6 years of professional experience in data engineering, data architecture, or a related role.

- Hands-on experience with distributed computing frameworks such as Hadoop, Spark, or Kafka.

- Strong proficiency in SQL and experience with database management systems (e., PostgreSQL, MySQL, or Oracle).

- Expertise in at least one programming language (Python, Scala, or Java).

- Familiarity with version control tools like Git and CI/CD pipelines.

- Experience with cloud services like AWS (S3, EMR, Glue), Azure (Data Factory, Synapse), or GCP (BigQuery, Dataflow).

Soft Skills :

- Excellent problem-solving abilities and analytical skills.

- Strong communication skills to interact effectively with technical and non-technical stakeholders.

Preferred Qualifications :

- Certification in cloud technologies (AWS Certified Data Analytics, Google Professional Data Engineer, or Azure Data Engineer Associate).

- Knowledge of big data tools like Apache Hive, Presto, or Airflow.

- Experience with data visualization tools like Tableau or Power BI