Data Engineer - SQL/PySpark

Job Title : Data Engineer (SQL, PySpark, Cloud)

Job Description :

We are seeking a skilled Data Engineer with strong experience in SQL, PySpark, and cloud technologies to join our dynamic team. The ideal candidate will have a solid background in designing and implementing data engineering pipelines, with a focus on performance, scalability, and reliability. You will work closely with other data engineers, data scientists, and stakeholders to develop, maintain, and optimize data infrastructure.

Key Responsibilities :

SQL Development :

- Write medium-complexity SQL queries to extract, transform, and load data efficiently.

- Optimize SQL queries for performance and scalability.

- Collaborate with team members to understand data requirements and translate them into actionable SQL scripts.

PySpark/Spark Development :

- Develop and maintain data processing scripts using PySpark or Spark with Scala.

- Implement data transformation and ETL processes using Spark to handle large-scale data.

- Optimize Spark jobs for performance and troubleshoot any issues that arise during processing.

Data Engineering Pipeline Design :

- Design and implement ETL/data engineering pipelines to support business intelligence and analytics needs.

- Ensure data pipelines are scalable, reliable, and maintainable.

- Collaborate with the team to understand data flow requirements and implement appropriate solutions.

Cloud Technology Implementation :

- Work with various cloud platforms (AWS, Azure, GCP) to build and manage data solutions.

- Implement cloud-based data storage and processing solutions, ensuring security and compliance.

- Utilize cloud services for data integration, orchestration, and processing.

Issue Resolution and Debugging :

- Identify, diagnose, and resolve data-related issues and bugs independently.

- Perform root cause analysis and implement fixes for any data discrepancies or processing failures.

- Monitor data pipelines and systems to ensure continuous availability and reliability.

CI/CD Integration :

- Implement and maintain basic to intermediate level CI/CD processes for data engineering projects.

- Collaborate with DevOps teams to ensure smooth deployment and integration of data solutions.

- Automate testing and deployment processes to improve development efficiency.

Desired Skills :

Data Modeling :

- Experience in designing data models to support analytical and business requirements.

Airflow :

- Experience with Apache Airflow for workflow scheduling and orchestration.

Kafka :

- Knowledge of Kafka for real-time data streaming and integration.

EMR/Databricks :

- Experience with AWS EMR or Databricks for big data processing and analytics.

Snowflake :

- Familiarity with Snowflake for cloud-based data warehousing solutions.

Hive :

- Understanding of Hive for querying and managing large datasets.

Qualifications :

Experience : 5+ years of experience in data engineering or related fields.

Technical Skills :

- Strong proficiency in SQL and writing medium-complexity queries.

- Hands-on experience with PySpark or Spark with Scala.

- Understanding of ETL processes and data engineering pipeline design.

- Basic to intermediate knowledge of CI/CD processes.

- Experience with cloud technologies (AWS, Azure, GCP).

Problem-Solving :

- Ability to troubleshoot and resolve data issues independently.

Communication :

- Strong verbal and written communication skills, with the ability to collaborate effectively with technical and non-technical stakeholders.

Education :

- Bachelor's degree in Computer Science, Information Technology, or related field.

Location : - Anywhere india ,Multiple Locations