Searce Cosourcing Services Senior Data Engineer Interview Questions and Answers

Q1. What is Databricks, and how does it differ from Azure Data Factory?
Add your answer

It was pair programming round where we need to attempt a couple of Spark Scenario along with the Interviewer. You will have a boiler plate code with some functionalities to be filled up. You will be assessed on writing clean and extensible code and test cases.

Q1. Questions will be based on the Project where they try to assess the depth of understading you have on your project, data quality checks, handling failure scenarios, reasoning behind choice of tech stack
Add your answer

Q1. About spark, data factory, Databricks etc
Add your answer
Q2. Need explain about these tools and technologies
Add your answer

Q1. SQL questions related to Data Engineering
Add your answer

Q1. SQL, Python and Data Engineering questions
Add your answer

Q1. Sql , python basics of spark
Add your answer

Q1. Complete SQL in depth
Ans. 
SQL is a programming language used for managing and manipulating relational databases.
• SQL stands for Structured Query Language

• It is used to create, modify, and query databases

• Common SQL commands include SELECT, INSERT, UPDATE, and DELETE

• SQL can be used with various database management systems such as MySQL, Oracle, and SQL Server
Answered by AI
Add your answer

Q1. What are accumulators in spark?
Ans. 
Accumulators are shared variables that are updated by worker nodes and can be used for aggregating information across tasks.
• Accumulators are used for implementing counters and sums in Spark.

• They are only updated by worker nodes and are read-only by the driver program.

• Accumulators are useful for debugging and monitoring purposes.

• Example: counting the number of errors encountered during processing.
Answered by AI
Add your answer
Q2. Explain spark architecture
Ans. 
Spark architecture is a distributed computing framework that consists of a driver program, cluster manager, and worker nodes.
• Spark architecture includes a driver program that manages the execution of the Spark application.

• It also includes a cluster manager that allocates resources and schedules tasks on worker nodes.

• Worker nodes are responsible for executing the tasks and storing data in memory or disk.

• Spark architectu...read more
Answered by AI
Add your answer
Q3. Write a query to find duplicate data using SQL
Ans. 
Query to find duplicate data using SQL
• Use GROUP BY and HAVING clause to identify duplicate records

• Select columns to check for duplicates

• Use COUNT() function to count occurrences of each record
Answered by AI
Add your answer

Q1. What is pub/sub?
Ans. 
Pub/sub is a messaging pattern where senders (publishers) of messages do not program the messages to be sent directly to specific receivers (subscribers).
• Pub/sub stands for publish/subscribe.

• Publishers send messages to a topic, and subscribers receive messages from that topic.

• It allows for decoupling of components in a system, enabling scalability and flexibility.

• Examples include Apache Kafka, Google Cloud Pub/Sub, and
Answered by AI
Add your answer
Q2. What services have you used in GCP
Ans. 
I have used services like BigQuery, Dataflow, Pub/Sub, and Cloud Storage in GCP.
• BigQuery for data warehousing and analytics

• Dataflow for real-time data processing

• Pub/Sub for messaging and event ingestion

• Cloud Storage for storing data and files
Answered by AI
Add your answer

1 hour, in your platform of choice. Must know OOPS concepts, TDD, Debugging, SOLID principles

Q1. 1. Different Types of integration runtime in adf 2. How to copy 100 files from one adls path to another in adf 3. Diff between DAG and Lineage , narrow and wide transformation in Spark 4. DBUtils questions...read more
Ans. 
The interview questions cover topics related to Azure Data Factory, Spark, and Python programming.
• Integration runtimes in ADF include Azure, Self-hosted, and SSIS IRs.

• To copy 100 files in ADF, use a Copy Data activity with a wildcard path in source and sink datasets.

• DAG in Spark represents a directed acyclic graph of computation, while lineage tracks the data flow.

• Narrow transformations in Spark operate on a single par...read more
Answered by AI
Add your answer

Q1. SQL and ADF concepts and PySpark project explanation
Add your answer