Evalueserve Senior Data Engineer Interview Questions and Answers

Q1. What is Databricks, and how does it differ from Azure Data Factory?
Add your answer

It was pair programming round where we need to attempt a couple of Spark Scenario along with the Interviewer. You will have a boiler plate code with some functionalities to be filled up. You will be assessed on writing clean and extensible code and test cases.

Q1. Questions will be based on the Project where they try to assess the depth of understading you have on your project, data quality checks, handling failure scenarios, reasoning behind choice of tech stack
Add your answer

Python and sql questions

Q1. About spark, data factory, Databricks etc
Add your answer
Q2. Need explain about these tools and technologies
Add your answer

Q1. Hadoop spark, SQL scenario based questions on windows functions and hive questions
Add your answer
Q2. More about spark and SQL
Add your answer

Q1. Architecture of spark
Ans. 
Spark is a distributed computing framework that provides in-memory processing capabilities for big data analytics.
• Spark has a master-slave architecture with a central coordinator called the Driver and distributed workers called Executors.

• It uses Resilient Distributed Datasets (RDDs) for fault-tolerant distributed data processing.

• Spark supports various data sources like HDFS, Cassandra, HBase, and S3 for input and outpu...read more
Answered by AI
Add your answer
Q2. SQL code for situations
Ans. 
SQL code for handling various situations in data analysis
• Use CASE statements for conditional logic

• Use COALESCE function to handle NULL values

• Use GROUP BY and HAVING clauses for aggregating data

• Use subqueries for complex filtering or calculations
Answered by AI
Add your answer

Q1. Manager round- describe about self and where do you see yourself in 5 years
Add your answer

Q1. Create spark dataframe
Ans. 
To create a Spark DataFrame, use the createDataFrame() method.
• Import the necessary libraries

• Create a list of tuples or a dictionary containing the data

• Create a schema for the DataFrame

• Use the createDataFrame() method to create the DataFrame
Answered by AI
Add your answer

Q1. What are accumulators in spark?
Ans. 
Accumulators are shared variables that are updated by worker nodes and can be used for aggregating information across tasks.
• Accumulators are used for implementing counters and sums in Spark.

• They are only updated by worker nodes and are read-only by the driver program.

• Accumulators are useful for debugging and monitoring purposes.

• Example: counting the number of errors encountered during processing.
Answered by AI
Add your answer
Q2. Explain spark architecture
Ans. 
Spark architecture is a distributed computing framework that consists of a driver program, cluster manager, and worker nodes.
• Spark architecture includes a driver program that manages the execution of the Spark application.

• It also includes a cluster manager that allocates resources and schedules tasks on worker nodes.

• Worker nodes are responsible for executing the tasks and storing data in memory or disk.

• Spark architectu...read more
Answered by AI
Add your answer
Q3. Write a query to find duplicate data using SQL
Ans. 
Query to find duplicate data using SQL
• Use GROUP BY and HAVING clause to identify duplicate records

• Select columns to check for duplicates

• Use COUNT() function to count occurrences of each record
Answered by AI
Add your answer

Q1. What is pub/sub?
Ans. 
Pub/sub is a messaging pattern where senders (publishers) of messages do not program the messages to be sent directly to specific receivers (subscribers).
• Pub/sub stands for publish/subscribe.

• Publishers send messages to a topic, and subscribers receive messages from that topic.

• It allows for decoupling of components in a system, enabling scalability and flexibility.

• Examples include Apache Kafka, Google Cloud Pub/Sub, and
Answered by AI
Add your answer
Q2. What services have you used in GCP
Ans. 
I have used services like BigQuery, Dataflow, Pub/Sub, and Cloud Storage in GCP.
• BigQuery for data warehousing and analytics

• Dataflow for real-time data processing

• Pub/Sub for messaging and event ingestion

• Cloud Storage for storing data and files
Answered by AI
Add your answer