SAVE Housing Finance Interview Questions and Answers

Delta Lake provides ACID transactions, schema enforcement, and time travel capabilities for data lakes.

• ACID transactions ensure data consistency and reliability.

• Schema enforcement helps maintain data quality and prevent data corruption.

• Time travel allows users to access and revert to previous versions of data for auditing or analysis purposes.

Filter students with marks greater than 80 in all subjects

• Iterate through each student's marks in all subjects

• Check if all marks are greater than 80 for a student

• Return the student if all marks are greater than 80

Syntax to define schema of a file for loading

• Use CREATE EXTERNAL TABLE statement in SQL

• Specify column names and data types in the schema definition

• Example: CREATE EXTERNAL TABLE MyTable (col1 INT, col2 STRING) USING CSV

Performance tuning in Azure Data Factory involves optimizing data flows and activities to improve efficiency and reduce processing time.

• Identify bottlenecks in data flows and activities

• Optimize data partitioning and distribution

• Use appropriate data integration patterns

• Leverage caching and parallel processing

• Monitor and analyze performance metrics

Azure Synapse is a cloud-based analytics service that brings together big data and data warehousing.

• Azure Synapse integrates big data and data warehousing capabilities in a single service

• It allows for data ingestion, preparation, management, and serving for BI and machine learning

• Supports both serverless and provisioned resources for data processing

• Offers integration with Azure Machine Learning, Power BI, and Azure Data Factory

Types of clusters in Databricks include Standard, High Concurrency, and Single Node clusters.

• Standard cluster: Suitable for running single jobs or workflows.

• High Concurrency cluster: Designed for multiple users running concurrent jobs.

• Single Node cluster: Used for development and testing purposes.

The catalyst optimizer is a query optimization engine in Apache Spark that improves performance by generating optimized query plans.

• It is a query optimization engine in Apache Spark.

• It improves performance by generating optimized query plans.

• It uses rule-based and cost-based optimization techniques.

• It leverages advanced techniques like code generation and adaptive query execution.

• Example: Catalyst optimizer in Spark SQL analyzes the query and generates an optimized query plan...read more

Catalyst optimizer is a query optimization framework in Apache Spark.

• Catalyst optimizer is a rule-based optimization framework used in Apache Spark for optimizing query plans.

• It leverages advanced programming language features in Scala to build an extensible query optimizer.

• Catalyst optimizer performs various optimizations such as constant folding, predicate pushdown, and projection pruning.

• It helps in improving the performance of Spark SQL queries by generating efficient que...read more