Accend Systems Data Engineer Interview Questions and Answers

Q1. What is the architecture of Apache Spark?
Ans. 
Apache Spark architecture includes a cluster manager, worker nodes, and driver program.
• Apache Spark architecture consists of a cluster manager, which allocates resources and schedules tasks.

• Worker nodes execute tasks and store data in memory or disk.

• Driver program coordinates tasks and communicates with the cluster manager.

• Spark applications run as independent sets of processes on a cluster, coordinated by the SparkCon...read more
Answered by AI
Add your answer
Q2. What is the difference between the reduceBy and groupBy transformations in Apache Spark?
Ans. 
reduceBy is used to aggregate data based on key, while groupBy is used to group data based on key.
• reduceBy is a transformation that combines the values of each key using an associative function and a neutral 'zero value'.

• groupBy is a transformation that groups the data based on a key and returns a grouped data set.

• reduceBy is more efficient for aggregating data as it reduces the data before shuffling, while groupBy shu...read more
Answered by AI
Add your answer
Q3. What is the difference between RDD (Resilient Distributed Datasets) and DataFrame in Apache Spark?
Ans. 
RDD is a low-level abstraction representing a distributed collection of objects, while DataFrame is a higher-level abstraction representing a distributed collection of data organized into named columns.
• RDD is more suitable for unstructured data and low-level transformations, while DataFrame is more suitable for structured data and high-level abstractions.

• DataFrames provide optimizations like query optimization and code...read more
Answered by AI
Add your answer
Q4. What are the different modes of execution in Apache Spark?
Ans. 
The different modes of execution in Apache Spark include local mode, standalone mode, YARN mode, and Mesos mode.
• Local mode: Spark runs on a single machine with one executor.

• Standalone mode: Spark runs on a cluster managed by a standalone cluster manager.

• YARN mode: Spark runs on a Hadoop cluster using YARN as the resource manager.

• Mesos mode: Spark runs on a Mesos cluster with Mesos as the resource manager.
Answered by AI
Add your answer

Q1. Sql order of execution
Ans. 
SQL order of execution determines the sequence in which different clauses are processed in a query.
• SQL query is parsed and validated first

• Next, the query optimizer creates an execution plan

• Execution plan includes steps like table scans, index scans, joins, etc.

• Finally, the query is executed and results are returned
Answered by AI
Add your answer
Q2. Sql question on finding highest salary for each employee where employees and department table are given in each department.
Add your answer

They asked various questions on guestimates and few SQL Questions

Q1. They asked questions on spark, SQL and Python
Add your answer
Q2. They asked questions on ADF as well
Add your answer

Q1. They majorly focused on PySpark and ADF in this round
Add your answer
Q2. They focused on the project also in this round
Add your answer

Q1. They focused on managerial-related questions in this round
Add your answer

Q1. How to Ensure Data loss in ETL pipeline
Add your answer
Q2. Why to spin up Dataproc cluster when there is serverless batch job exists.
Add your answer

Aptitude test involved with quantative aptitude, logical reasoning and reading comprehensions.

Q1. Tell me your introduction.
Add your answer
Q2. Tell me about your skills.
Ans. 
I have strong skills in data processing, ETL, data modeling, and programming languages like Python and SQL.
• Proficient in data processing and ETL techniques

• Strong knowledge of data modeling and database design

• Experience with programming languages like Python and SQL

• Familiarity with big data technologies such as Hadoop and Spark
Answered by AI
Add your answer

Q1. Are you ready relocate,?
Ans. 
Yes, I am open to relocating for the right opportunity.
• I am willing to relocate for the right job opportunity.

• I have experience moving for previous roles.

• I am flexible and adaptable to new locations.

• I am excited about the possibility of exploring a new city or country.
Answered by AI
Add your answer
Q2. Document verification
Add your answer

Q1. List vs tuple ?
Add your answer
Q2. What is Slowly changing dimension
Add your answer

Q1. What is Databricks, Internals of it, Optimization technique
Ans. 
Databricks is a unified analytics platform that provides a collaborative environment for data scientists, engineers, and analysts.
• Databricks is built on top of Apache Spark, providing a unified platform for data engineering, data science, and business analytics.

• Internals of Databricks include a cluster manager, job scheduler, and workspace for collaboration.

• Optimization techniques in Databricks include query optimizati...read more
Answered by AI
Add your answer
Q2. SQL questions of joins,group by
Add your answer

Q1. Scenario based azure data factory questions
Add your answer
Q2. Project structres, pyspark dataframes related
Add your answer

Its mandatory test even for experience people

Q1. Related to technology
Add your answer

Q1. Very good discussion towards work culture, salary and all
Add your answer