NTT Data Big Data Engineer Interview Questions and Answers

Q1. All type of joins with example
Ans. 
Different types of joins in SQL with examples
• Inner Join: Returns rows when there is a match in both tables

• Left Join: Returns all rows from the left table and the matched rows from the right table

• Right Join: Returns all rows from the right table and the matched rows from the left table

• Full Outer Join: Returns all rows when there is a match in either table
Answered by AI
Add your answer
Q2. How to handle large spark datasets
Ans. 
Large Spark datasets can be handled by partitioning, caching, optimizing transformations, and tuning resources.
• Partitioning data to distribute workload evenly across nodes

• Caching frequently accessed data to avoid recomputation

• Optimizing transformations to reduce unnecessary processing

• Tuning resources like memory allocation and parallelism for optimal performance
Answered by AI
Add your answer

Q1. How do you tune sparks configuration setting's to optimize query performance
Ans. 
Spark configuration settings can be tuned to optimize query performance by adjusting parameters like memory allocation, parallelism, and caching.
• Increase executor memory and cores to allow for more parallel processing

• Adjust shuffle partitions to optimize data shuffling during joins and aggregations

• Enable dynamic allocation to scale resources based on workload demands

• Utilize caching to store intermediate results and avo...read more
Answered by AI
Add your answer
Q2. What strategies do you use to handle data skew and partition imbalance in spark
Ans. 
To handle data skew and partition imbalance in Spark, strategies include using salting, bucketing, repartitioning, and optimizing join operations.
• Use salting to evenly distribute skewed keys across partitions

• Implement bucketing to pre-partition data based on a specific column

• Repartition data based on a specific key to balance partitions

• Optimize join operations by broadcasting small tables or using partitioning strategi
Answered by AI
Add your answer

Q1. Big Data fundamentals concepts on rdd, dataframe and datasets.
Add your answer
Q2. Spark optimization techniques
Ans. 
Spark optimization techniques involve partitioning, caching, and tuning resource allocation.
• Partitioning data to distribute workload evenly

• Caching frequently accessed data to avoid recomputation

• Tuning resource allocation for optimal performance
Answered by AI
Add your answer
Q3. SQL queries on windows function and joins.
Add your answer

30 Questions in 20 Minutes

Q1. Baiscs of SQL,Python,AWS and spark in depth question
Add your answer

More questions about coding in sql & pyspark

Q1. Coalesce vs repartition
Ans. 
Coalesce is used to reduce the number of partitions in a DataFrame, while repartition is used to increase the number of partitions.
• Coalesce is a narrow transformation that can only decrease the number of partitions without shuffling data.

• Repartition is a wide transformation that can both increase or decrease the number of partitions and involves shuffling data across the cluster.

• Coalesce is more efficient for reducing ...read more
Answered by AI
Add your answer

Rank vs dense rank quetions ctes

Python data structure

Q1. Scenario based questions
Add your answer

Q1. Project explanation , internal vs external table
Ans. 
Internal tables are managed by Hive, while external tables are managed by the user.
• Internal tables are stored in a Hive-managed warehouse directory, while external tables can be stored anywhere.

• Internal tables are deleted when the table is dropped, while external tables are not.

• External tables can be used to access data stored in non-Hive formats, such as CSV or JSON.

• Internal tables are typically used for temporary or ...read more
Answered by AI
Add your answer

Q1. What is your current salary and your expectations
Add your answer

Q1. Spark optimization techniques
Ans. 
Spark optimization techniques improve performance and efficiency of Spark applications.
• Partitioning data to reduce shuffling

• Caching frequently used data

• Using broadcast variables for small data

• Using efficient data formats like Parquet

• Tuning memory and CPU usage

• Using appropriate cluster size

• Avoiding unnecessary data shuffling

• Using appropriate serialization formats

• Using appropriate join strategies
Answered by AI
Add your answer
Q2. What type of filesystem used in ur project
Ans. 
We use Hadoop Distributed File System (HDFS) for our project.
• HDFS is a distributed file system designed to run on commodity hardware.

• It provides high-throughput access to application data and is fault-tolerant.

• HDFS is used by many big data processing frameworks like Hadoop, Spark, etc.

• It stores data in a distributed manner across multiple nodes in a cluster.

• HDFS is optimized for large files and sequential reads and wri
Answered by AI
Add your answer
Q3. Spark + SQL basic coding
Add your answer