AmbitionBox
Search
Jobs
- Reviews
- Salaries
- Interview Questions
- About Company
- Benefits
- Jobs
- Office Photos
- Community
Add office photos
All Filters
1 Data Economy Senior Data Engineer Job
Senior Data Engineer
Data Economy
posted 3d ago
Job Role Insights
Flexible timing
Key skills for the job
Job Description
Job Title: Senior Data Engineer - Data bricks and PySpark on AWS
Location: Hyderabad/Pune.
Experience: 5-10 Years
Employment Type: Full-Time
Job Description:
We are seeking a Senior Data Engineer with 5-10 years of experience to join our dynamic team. The ideal candidate will be responsible for designing, building, and optimizing big data pipelines using Data bricks and PySpark on AWS. You will work closely with data scientists, analysts, and other engineers to transform raw data into actionable insights, supporting critical business decisions.
Key Responsibilities:
1.Data Engineering & ETL Pipelines:
-Design, develop, and optimize scalable ETL pipelines using Data bricks and PySpark.
-Process structured and unstructured data from various sources such as Amazon S3, Delta Lake, and relational databases.
2.Big Data Processing:
-Utilize PySpark to process and analyze large datasets efficiently.
-Implement distributed computing solutions to solve complex data problems.
3.Cloud Integration:
-Work extensively with AWS services like S3, Glue, Redshift, Lambda, and EMR to build and manage data pipelines.
-Integrate Data bricks workflows with the broader AWS ecosystem.
4.Delta Lake & Storage Optimization:
-Design and implement Delta Lake solutions to ensure data reliability, scalability, and performance.
-Optimize data storage for faster querying and analytics.
5.Job Scheduling & Automation:
-Schedule and monitor data pipelines using Data bricks jobs or AWS-based orchestration tools.
6.Collaboration & Best Practices:
-Collaborate with data scientists and analysts to understand business needs and deliver data solutions.
-Enforce coding best practices and performance optimization techniques for PySpark and Data bricks workflows.
7.Monitoring & Troubleshooting:
-Monitor job performance, troubleshoot failures, and optimize cluster usage to ensure cost-effectiveness.
Requirements
-Technical Expertise:
-5 to 10 years of experience with Data bricks and Apache Spark, preferably with PySpark.
-Strong experience with AWS services like S3, Glue, Redshift, Lambda, and EMR.
-Proficiency in programming with Python and SQL.
-Expertise in Delta Lake and data lake management.
-Knowledge of job scheduling and orchestration tools (e.g., Data bricks Jobs, Airflow).
-Data Processing:
-Hands-on experience with distributed data processing, streaming, and batch data pipelines.
-Proficient in working with large datasets and implementing performance-optimized solutions.
-Soft Skills:
-Excellent problem-solving and analytical skills.
-Strong communication and collaboration abilities.
-Ability to work independently and in cross-functional teams.
Preferred Skills:
-Experience with machine learning workflows and integrating with tools like SageMaker or MLflow.
-Familiarity with Kubernetes and containerized Spark clusters.
-Knowledge of data governance frameworks and compliance standards like GDPR or HIPAA.
Education:
-Bachelor\u2019s or Master\u2019s degree in Computer Science, Data Engineering, or a related field.
Benefits
Location: Hyderabad/Pune.
Experience: 5-10 Years
Employment Type: Full-Time
Job Description:
We are seeking a Senior Data Engineer with 5-10 years of experience to join our dynamic team. The ideal candidate will be responsible for designing, building, and optimizing big data pipelines using Data bricks and PySpark on AWS. You will work closely with data scientists, analysts, and other engineers to transform raw data into actionable insights, supporting critical business decisions.
Key Responsibilities:
1.Data Engineering & ETL Pipelines:
-Design, develop, and optimize scalable ETL pipelines using Data bricks and PySpark.
-Process structured and unstructured data from various sources such as Amazon S3, Delta Lake, and relational databases.
2.Big Data Processing:
-Utilize PySpark to process and analyze large datasets efficiently.
-Implement distributed computing solutions to solve complex data problems.
3.Cloud Integration:
-Work extensively with AWS services like S3, Glue, Redshift, Lambda, and EMR to build and manage data pipelines.
-Integrate Data bricks workflows with the broader AWS ecosystem.
4.Delta Lake & Storage Optimization:
-Design and implement Delta Lake solutions to ensure data reliability, scalability, and performance.
-Optimize data storage for faster querying and analytics.
5.Job Scheduling & Automation:
-Schedule and monitor data pipelines using Data bricks jobs or AWS-based orchestration tools.
6.Collaboration & Best Practices:
-Collaborate with data scientists and analysts to understand business needs and deliver data solutions.
-Enforce coding best practices and performance optimization techniques for PySpark and Data bricks workflows.
7.Monitoring & Troubleshooting:
-Monitor job performance, troubleshoot failures, and optimize cluster usage to ensure cost-effectiveness.
Requirements
-Technical Expertise:
-5 to 10 years of experience with Data bricks and Apache Spark, preferably with PySpark.
-Strong experience with AWS services like S3, Glue, Redshift, Lambda, and EMR.
-Proficiency in programming with Python and SQL.
-Expertise in Delta Lake and data lake management.
-Knowledge of job scheduling and orchestration tools (e.g., Data bricks Jobs, Airflow).
-Data Processing:
-Hands-on experience with distributed data processing, streaming, and batch data pipelines.
-Proficient in working with large datasets and implementing performance-optimized solutions.
-Soft Skills:
-Excellent problem-solving and analytical skills.
-Strong communication and collaboration abilities.
-Ability to work independently and in cross-functional teams.
Preferred Skills:
-Experience with machine learning workflows and integrating with tools like SageMaker or MLflow.
-Familiarity with Kubernetes and containerized Spark clusters.
-Knowledge of data governance frameworks and compliance standards like GDPR or HIPAA.
Education:
-Bachelor\u2019s or Master\u2019s degree in Computer Science, Data Engineering, or a related field.
Benefits
What We Offer:
-Opportunity to work with cutting-edge technologies in a collaborative environment.
-Continuous learning and professional development opportunities.
","
-Opportunity to work with cutting-edge technologies in a collaborative environment.
-Continuous learning and professional development opportunities.
","
Employment Type: Full Time, Permanent
Read full job descriptionPrepare for Senior Data Engineer roles with real interview advice
What Senior Data Engineer at Data Economy are saying
Rating based on 2 Senior Data Engineer
reviews
Anonymous · Data Science & Analytics
in Hyderabad / Secunderabad
Likes
Working on Latest Technologies. Learning Outcomes. Senior Management.
- Salary - Good
+6 more
Dislikes
No Additional Perks.
Senior Data Engineer salary at Data Economy
reported by
5
employees
with 3-5
years exp.
₹10
L/yr - ₹12
L/yr
40%
less
than the average Senior Data Engineer Salary in India
View more details
What Data Economy employees are saying about work life
based on 52 employees
Flexible timing
Monday to Friday
No travel
Day Shift
Data Economy Benefits
Work From Home 
Soft Skill Training 
Health Insurance 
Cafeteria 
Education Assistance 
Compare Data Economy with
TCS
Compare
Infosys
Compare
Wipro
Compare
HCLTech
Compare
Tech Mahindra
Compare
LTIMindtree
Compare
Mphasis
Compare
Hexaware Technologies
Compare
Cyient
Compare
JoulestoWatts Business Solutions
Compare
Thoughtsol Infotech
Compare
Innoplexus
Compare
NexTurn
Compare
Blackbuck Insights
Compare
Unify Technologies
Compare
Nineleaps Technology Solutions
Compare
Junglee Games
Compare
Metayb
Compare
Karza Technologies
Compare
Devon Software Services
Compare
Share an Interview