Data Engineer

Position: Data Engineer

Experience: 3 to 6 Yrs

Notice Period: Up to 1 Month

Job Summary:
We are seeking a highly motivated and skilled Data Engineer with atleast 3 years of experience in designing, building and optimizing data pipelines, designing data architectures, to process and transform large volumes of data from various sources and working with cloud platforms preferably AWS, and Databricks. The ideal candidate will have a strong background in data engineering, cloud-based solutions, and large-scale data processing.

Key Responsibilities:

• Design and Build Data Pipelines:
  Develop, test, and optimize scalable ETL/ELT workflows and data pipelines for batch and real-time data processing using tools like Databricks, Apache Spark, and cloud-native services.
• Cloud Data Solutions:
  Design and implement data solutions using cloud platforms such as AWS (e.g., S3, Glue, RDS, Lambda) or similar environments.
• Data Modeling and Storage:
  Create optimized data models and manage data storage solutions like Delta Lake, or other data lake.
• Data Quality and Governance:
  Implement data validation, quality checks, and monitoring to ensure data accuracy and compliance with security and governance standards.
• Collaborate with Cross-Functional Teams:
  Work closely with data scientists, analysts, and business stakeholders to gather requirements and deliver data solutions that meet business needs.
• Performance Optimization:
  Optimize query performance and storage to improve the efficiency of data processing systems.
• Monitoring and Troubleshooting:
  Set up monitoring tools and resolve issues related to data pipelines and cloud infrastructure.

Required Skills:

• Technical Skills:
• 3+ years of experience in data engineering or a related role.
• Proficiency in Python and PySpark for data manipulation and analysis and writing complex queries in SQL.
• Proficiency with distributed data frameworks such as Apache Spark and creating notebooks, jobs, and optimizing Delta Lake-based solutions in Databricks.
• Hands-on experience with cloud services (AWS, Azure, or GCP)

• Other Skills:
• Familiarity with CI/CD pipelines and version control systems like Git.
• Knowledge / familiarity with data visualization tools like Power BI / AWS Quicksight to create insightful and interactive reports.