Creditsafe Data Engineer Interview Questions and Answers for Experienced

Q1. What is the SQL query to calculate the average sales over a period of 7 days?
Add your answer
Q2. Explain Data bricks architecture?
Add your answer

Q1. Basic Python, SQL & Spark coding questions.
Add your answer
Q2. Nth highest salary, word count in pyspark
Ans. 
To find the nth highest salary in pyspark, use the window function with row_number and filter on the desired rank.
• Use window function with row_number to assign a rank to each salary

• Filter the result to get the row with the desired rank

• Example: df.withColumn('rank', F.row_number().over(Window.orderBy(F.col('salary').desc()))).filter(F.col('rank') == n).select('salary')
Answered by AI
Add your answer

Q1. Questions on SQL & PySpark.
Add your answer

Q1. Questions about previous projects & optimization techniques etc. Also behavioural
Add your answer

Q1. Salary negotiation part.
Add your answer

Q1. Explain the architecture of delta lake
Ans. 
Delta Lake is an open-source storage layer that brings ACID transactions to Apache Spark and big data workloads.
• Delta Lake provides ACID transactions, scalable metadata handling, and unifies streaming and batch data processing.

• It stores data in Parquet format and uses Apache Spark for processing.

• Delta Lake ensures data reliability and data quality by providing schema enforcement and data versioning.

• It supports time tra...read more
Answered by AI
Add your answer
Q2. Tuning operations in databricks
Ans. 
Tuning operations in Databricks involves optimizing performance and efficiency of data processing tasks.
• Use cluster configuration settings to allocate resources efficiently

• Optimize code by minimizing data shuffling and reducing unnecessary operations

• Leverage Databricks Auto Optimize to automatically tune performance

• Monitor job performance using Databricks Runtime Metrics and Spark UI
Answered by AI
Add your answer

Q1. Normal questions on spark, hadoop, sql
Add your answer
Q2. One python coding question was there
Add your answer

Q1. Python coding felt difficult, its a moderate difficult question
Add your answer
Q2. Questions on project, sql moderate questions
Add your answer

Q1. Spark architecture
Add your answer

Q1. Explain Spark RDD,DF
Ans. 
Spark RDD and DF are two data structures in Apache Spark for processing and analyzing data.
• RDD (Resilient Distributed Dataset) is a distributed collection of elements that can be operated on in parallel. It is immutable and fault-tolerant.

• DF (DataFrame) is a distributed collection of data organized into named columns. It provides a more structured and efficient way to work with data compared to RDDs.

• RDD is low-level an...read more
Answered by AI
Add your answer
Q2. Spark Architecture
Add your answer

End to end data engineering architecture

Q1. Salary expectation
Add your answer
Q2. Cultural fitment
Add your answer

Q1. Mostly on Spark
Add your answer

Q1. What is executor memory
Ans. 
Executor memory is the amount of memory allocated to each executor in a Spark application.
• Executor memory is specified using the 'spark.executor.memory' configuration property.

• It determines how much memory each executor can use to process tasks.

• It is important to properly configure executor memory to avoid out-of-memory errors or inefficient resource utilization.
Answered by AI
Add your answer
Q2. Difference between rdd and data frame
Ans. 
RDD is a low-level abstraction in Spark representing distributed data, while DataFrames are higher-level structured APIs for working with data.
• RDD is an immutable distributed collection of objects, while DataFrames are distributed collection of data organized into named columns.

• RDDs are more suitable for unstructured data and low-level transformations, while DataFrames provide a more user-friendly API for structured da...read more
Answered by AI
Add your answer