SG Analytics Data Engineer Interview Questions and Answers

Aws, scala and cloud concept

Q1. Python question
Add your answer

Q1. What is pip in python
Ans. 
pip is a package manager for Python that allows you to install and manage libraries and dependencies.
• pip is used to install Python packages from the Python Package Index (PyPI)

• It can also be used to uninstall packages, upgrade packages, and list installed packages

• Example: 'pip install numpy' installs the numpy package
Answered by AI
Add your answer
Q2. Questions on join
Add your answer

Basic aptitude questions

Q1. SQL basics questions like joins
Add your answer
Q2. Pandas basics for data manipulation
Add your answer

Test is based on python, sql and linux

Interview is based on projects

Q1. Spark arcitecture
Add your answer
Q2. Partitions and bucketing
Add your answer

Apptitude test happen in first go it was quite easy

GD happen between 10 group member we have speak for sometime

Q1. Question on OOPS
Add your answer
Q2. Quesion on SQL and python
Add your answer

Q1. Explain hadoop ecosystem
Ans. 
Hadoop ecosystem is a collection of open-source software tools used for distributed storage and processing of big data.
• Hadoop Distributed File System (HDFS) for storage

• MapReduce for processing

• Apache Hive for data warehousing

• Apache Pig for data analysis

• Apache Spark for real-time processing

• Apache HBase for NoSQL database

• Apache Kafka for real-time data streaming
Answered by AI
Add your answer
Q2. Explain YARN architecture
Ans. 
YARN architecture is a resource management layer in Hadoop ecosystem for job scheduling and cluster resource management.
• YARN stands for Yet Another Resource Negotiator.

• It consists of ResourceManager, NodeManager, ApplicationMaster, and Container.

• ResourceManager manages resources and schedules applications.

• NodeManager runs on each node and manages resources available on that node.

• ApplicationMaster negotiates resources w...read more
Answered by AI
Add your answer
Q3. Hive external and internal tables
Add your answer
Q4. Sort by and order by difference
Ans. 
Sort by is used to arrange data in a specific order, while order by is used to specify the sorting order in SQL queries.
• Sort by is used in programming languages to arrange data based on a specific criteria

• Order by is used in SQL queries to specify the sorting order of the result set

• Sort by is more commonly used in programming languages like Python, while order by is used in SQL queries
Answered by AI
Add your answer
Q5. Broadcast and accumulator
Add your answer
Q6. Optimization techniques
Ans. 
Optimization techniques are methods used to improve the efficiency and performance of data processing.
• Use indexing to speed up data retrieval

• Implement caching to reduce redundant computations

• Utilize parallel processing for faster execution

• Optimize algorithms for better performance

• Use data compression techniques to reduce storage space
Answered by AI
Add your answer

Q1. It was a managerial round which was mostly around the project work and working experience in a team
Add your answer
Q2. What was your role and responsisbility in the project
Add your answer
Q3. Why is the approach of the project in this manner
Add your answer
Q4. Few scenario based question
Add your answer

Q1. Compensation and salary discussion
Add your answer