Virtusa Consulting Services Big Data Engineer Interview Questions, Process, and Tips

Q1. Basic Questions of Scala Functional Programming concepts.
Add your answer
Q2. Spark internal working and optimization techniques
Ans. 
Spark internal working and optimization techniques
• Spark uses Directed Acyclic Graph (DAG) for optimizing workflows

• Lazy evaluation helps in optimizing transformations by combining them into a single stage

• Caching and persistence of intermediate results can improve performance

• Partitioning data can help in parallel processing and reducing shuffle operations
Answered by AI
Add your answer

The aptitude test lasts 30 minutes and focuses on topics relevant to data engineering, including Spark, SQL, Azure, and PySpark.

The coding test is a one-hour examination on PySpark.

Q1. What is the difference between Cache() and Persist()?
Add your answer
Q2. What does the purpose of the Spark Submit command in Apache Spark?
Add your answer
Q3. What are window functions in SQL?
Add your answer

Q1. Could you provide more details about the daily responsibilities associated with this role?
Add your answer
Q2. How would you describe your work culture?
Add your answer

Q1. How can you design an Azure Data Factory pipeline to copy data from a folder containing files with different delimiters to another folder?
Add your answer
Q2. Write a pyspark program that reads multiple csv files and creates a data frame with count of records against each file
Add your answer

Q1. DAX Related Syntax and Codes
Add your answer
Q2. Data Modelling, SQL, Python
Add your answer

Q1. No Response from HR after calling of selection after Round 1
Add your answer

Q1. Windows function-related questions.
Add your answer
Q2. Join Related Questions.
Add your answer

Q1. Join related Questions
Add your answer
Q2. Subqueries related queations.
Add your answer

Q1. How do you utilize the enhanced optimization option in AWS Glue?
Ans. 
Enhanced optimization in AWS Glue improves job performance by automatically adjusting resources based on workload
• Enhanced optimization in AWS Glue automatically adjusts resources like DPUs based on workload

• It helps improve job performance by optimizing resource allocation

• Users can enable enhanced optimization in AWS Glue job settings
Answered by AI
Add your answer
Q2. What are the best practices for optimizing querying in Amazon Redshift?
Ans. 
Optimizing querying in Amazon Redshift involves proper table design, distribution keys, sort keys, and query optimization techniques.
• Use appropriate distribution keys to evenly distribute data across nodes for parallel processing.

• Utilize sort keys to physically order data on disk, reducing the need for sorting during queries.

• Avoid using SELECT * and instead specify only the columns needed to reduce data transfer.

• Use AN...read more
Answered by AI
Add your answer

Q1. What is scd type 2?
Ans. 
SCD type 2 is a method used in data warehousing to track historical changes by creating a new record for each change.
• SCD type 2 stands for Slowly Changing Dimension type 2

• It involves creating a new record in the dimension table whenever there is a change in the data

• The old record is marked as inactive and the new record is marked as current

• It allows for historical tracking of changes in data over time

• Example: If a cust...read more
Answered by AI
Add your answer
Q2. Pyspark question read CSV from folder and add column in each csv file and write it to different location.
Add your answer