Celebal Technologies Senior Data Engineer Interview Questions and Answers

Q1. Cache vs persist
Ans. 
Cache is used for temporary storage of data in memory, while persist is used for saving data to disk for long-term storage.
• Cache is typically faster as it stores data in memory for quick access.

• Persist saves data to disk for durability and long-term storage.

• Cache is often used for temporary data that can be recomputed if lost, while persist is used for important data that needs to be retained.

• Examples: Using cache for ...read more
Answered by AI
Add your answer
Q2. Broadcast join and other optimization techniques
Add your answer

Q1. Sql window function
Add your answer
Q2. Spark architecture, dataframe vs rdd
Add your answer
Q3. Python reverse sentence
Ans. 
Reverse a sentence using Python
• Split the sentence into words using split() method

• Reverse the list of words using list slicing

• Join the reversed list of words back into a sentence using join() method
Answered by AI
Add your answer
Q4. Join based on nulls in sql
Add your answer

Q1. How did you ingest kafka data and you day to day activity
Ans. 
I ingested Kafka data using Kafka Connect and performed data processing and analysis on a daily basis.
• Used Kafka Connect to ingest data from various sources into Kafka topics

• Developed custom Kafka Connect connectors for specific data sources

• Utilized Kafka Streams for real-time data processing and analysis

• Worked with schema registry for data serialization and deserialization
Answered by AI
Add your answer
Q2. What is catalyst optimizer
Ans. 
Catalyst optimizer is a query optimization framework in Apache Spark that improves performance by generating optimized query plans.
• Catalyst optimizer is a rule-based optimization framework in Apache Spark.

• It leverages advanced techniques like predicate pushdown, constant folding, and join reordering to optimize query plans.

• Catalyst optimizer helps in improving the performance of Spark SQL queries by selecting the most ...read more
Answered by AI
Add your answer

SQL test, 3-5 questions. Easy to moderate difficult.

Q1. Your current and expected slaray
Add your answer
Q2. Do you have any offer in hand
Ans. 
Yes, I have received an offer from another company.
• Received an offer from XYZ company for a Senior Data Engineer position

• Currently evaluating the offer and considering my options

• Have not made a decision yet
Answered by AI
Add your answer

Q1. Design zepto delivery system .
Ans. 
Design a zepto delivery system for efficient and fast delivery of small packages.
• Utilize drones for quick and efficient delivery

• Implement a centralized hub for package sorting and distribution

• Use GPS tracking for real-time package monitoring

• Optimize routes for faster delivery times

• Ensure secure and reliable delivery process
Answered by AI
Add your answer

Logical questions which are highly useful

Multiple coding questions they asked in pyspark

Q1. Linux questions
Add your answer
Q2. Boot process of linux
Ans. 
The Linux boot process involves several stages, including BIOS/UEFI, bootloader, kernel initialization, and user space initialization.
• BIOS/UEFI performs hardware checks and loads bootloader

• Bootloader (GRUB, LILO) loads kernel into memory

• Kernel initializes hardware, mounts root filesystem, and starts user space processes

• User space initialization includes starting system services and login prompt
Answered by AI
Add your answer
Q3. Rhel6 vs rhel 7
Ans. 
RHEL 7 offers improved performance, security features, and support for newer hardware compared to RHEL 6.
• RHEL 7 has better performance due to kernel improvements and optimized resource management.

• RHEL 7 includes enhanced security features such as SELinux improvements and system-wide cryptographic policies.

• RHEL 7 provides support for newer hardware and technologies, such as Docker containers and systemd.

• RHEL 7 offers im...read more
Answered by AI
Add your answer

Q1. AWS questions about ec2, security groups, roles and etc
Add your answer

Q1. Salary discussion and other benefits
Add your answer

4 Coding Questions Leetcode Level Medium to Hard. Had to solve it within 1.5 hours.

Was able to solve 2 questions - One hard and one Medium. Hadn't enough time to solve all.

Q1. DSA based round. Was asked to write 3 APIs based on the the given n-ary tree.
Add your answer
Q2. Questions based on Path, end node, best path, etc.
Add your answer

Q1. System Design round : Was asked to design RDBMS for the given problem.
Add your answer

Q1. HM Round : Mostly behavioral based around Twilio Magic Values
Add your answer

Q1. Bar Raiser Round : Taken by a director. Was asked behavioral questions again.
Add your answer