Altimetrik Data Science Intern Interview Questions and Answers

Q1. How to create pipeline in adf?
Ans. 
To create a pipeline in ADF, you can use the Azure Data Factory UI or code-based approach.
• Use Azure Data Factory UI to visually create and manage pipelines

• Use code-based approach with JSON to define pipelines and activities

• Add activities such as data movement, data transformation, and data processing to the pipeline

• Set up triggers and schedules for the pipeline to run automatically
Answered by AI
Add your answer
Q2. Diffrent types of activities in pipelines
Ans. 
Activities in pipelines include data extraction, transformation, loading, and monitoring.
• Data extraction: Retrieving data from various sources such as databases, APIs, and files.

• Data transformation: Cleaning, filtering, and structuring data for analysis.

• Data loading: Loading processed data into a data warehouse or database.

• Monitoring: Tracking the performance and health of the pipeline to ensure data quality and reliab
Answered by AI
Add your answer
Q3. What is use of getmetadata
Ans. 
getmetadata is used to retrieve metadata information about a dataset or data source.
• getmetadata can provide information about the structure, format, and properties of the data.

• It can be used to understand the data schema, column names, data types, and any constraints or relationships.

• This information is helpful for data engineers to properly process, transform, and analyze the data.

• For example, getmetadata can be used ...read more
Answered by AI
Add your answer
Q4. Diffrent types of triggers
Ans. 
Triggers in databases are special stored procedures that are automatically executed when certain events occur.
• Types of triggers include: DML triggers (for INSERT, UPDATE, DELETE operations), DDL triggers (for CREATE, ALTER, DROP operations), and logon triggers.

• Triggers can be classified as row-level triggers (executed once for each row affected by the triggering event) or statement-level triggers (executed once for eac...read more
Answered by AI
Add your answer
Q5. Diffrence between normal cluster and job cluster in databricks
Ans. 
Normal cluster is used for interactive workloads while job cluster is used for batch processing in Databricks.
• Normal cluster is used for ad-hoc queries and exploratory data analysis.

• Job cluster is used for running scheduled jobs and batch processing tasks.

• Normal cluster is terminated after a period of inactivity, while job cluster is terminated after the job completes.

• Normal cluster is more cost-effective for short-liv...read more
Answered by AI
Add your answer
Q6. What is slowly changing dimensions
Ans. 
Slowly changing dimensions refer to data warehouse dimensions that change slowly over time.
• SCDs are used to track historical changes in data over time.

• There are three types of SCDs - Type 1, Type 2, and Type 3.

• Type 1 SCDs overwrite old data with new data, Type 2 creates new records for changes, and Type 3 maintains both old and new data in separate columns.

• Example: A customer's address changing would be a Type 2 SCD.

• Ex...read more
Answered by AI
Add your answer
Q7. Incremental load
Add your answer
Q8. With use in python
Ans. 
Use Python's 'with' statement to ensure proper resource management and exception handling.
• Use 'with' statement to automatically close files after use

• Helps in managing resources like database connections

• Ensures proper cleanup even in case of exceptions
Answered by AI
Add your answer
Q9. List vs tuple in python
Ans. 
List is mutable, tuple is immutable in Python.
• List can be modified after creation, tuple cannot be modified.

• List uses square brackets [], tuple uses parentheses ().

• Lists are used for collections of items that may need to be changed, tuples are used for fixed collections of items.

• Example: list_example = [1, 2, 3], tuple_example = (4, 5, 6)
Answered by AI
Add your answer
Q10. Datalake 1 vs datalake2
Ans. 
Datalake 1 and Datalake 2 are both storage systems for big data, but they may differ in terms of architecture, scalability, and use cases.
• Datalake 1 may use a Hadoop-based architecture while Datalake 2 may use a cloud-based architecture like AWS S3 or Azure Data Lake Storage.

• Datalake 1 may be more suitable for on-premise data storage and processing, while Datalake 2 may offer better scalability and flexibility for clou...read more
Answered by AI
Add your answer
Q11. How to read a file in databricks
Ans. 
To read a file in Databricks, you can use the Databricks File System (DBFS) or Spark APIs.
• Use dbutils.fs.ls('dbfs:/path/to/file') to list files in DBFS

• Use spark.read.format('csv').load('dbfs:/path/to/file') to read a CSV file

• Use spark.read.format('parquet').load('dbfs:/path/to/file') to read a Parquet file
Answered by AI
Add your answer
Q12. Star vs snowflake schema
Ans. 
Star schema is denormalized with one central fact table surrounded by dimension tables, while snowflake schema is normalized with multiple related dimension tables.
• Star schema is easier to understand and query due to denormalization.

• Snowflake schema saves storage space by normalizing data.

• Star schema is better for data warehousing and OLAP applications.

• Snowflake schema is better for OLTP systems with complex relationsh
Answered by AI
Add your answer
Q13. Repartition vs coalesece
Ans. 
repartition increases partitions while coalesce decreases partitions in Spark
• repartition shuffles data and can be used for increasing partitions for parallelism

• coalesce reduces partitions without shuffling data, useful for reducing overhead

• repartition is more expensive than coalesce as it involves data movement

• example: df.repartition(10) vs df.coalesce(5)
Answered by AI
Add your answer
Q14. Parquet file uses
Ans. 
Parquet file format is a columnar storage format used for efficient data storage and processing.
• Parquet files store data in a columnar format, which allows for efficient querying and processing of specific columns without reading the entire file.

• It supports complex nested data structures like arrays and maps.

• Parquet files are highly compressed, reducing storage space and improving query performance.

• It is commonly used ...read more
Answered by AI
Add your answer

Q1. What can you improve the query performance?
Ans. 
Improving query performance by optimizing indexes, using proper data types, and minimizing data retrieval.
• Optimize indexes on frequently queried columns

• Use proper data types to reduce storage space and improve query speed

• Minimize data retrieval by only selecting necessary columns

• Avoid using SELECT * in queries

• Use query execution plans to identify bottlenecks and optimize accordingly
Answered by AI
Add your answer
Q2. What id SCD type2 table?
Ans. 
SCD type2 table is used to track historical changes in data by creating new records for each change.
• Contains current and historical data

• New records are created for each change

• Includes effective start and end dates for each record

• Requires additional columns like surrogate keys and version numbers

• Used for slowly changing dimensions in data warehousing
Answered by AI
Add your answer

Q1. Azure Scenario based questions
Add your answer
Q2. Pyspark Coding based questions
Add your answer

Q1. ADF, Databricks related question
Add your answer
Q2. Spark Performance problem and scenarios
Ans. 
Spark performance problems can arise due to inefficient code, data skew, resource constraints, and improper configuration.
• Inefficient code can lead to slow performance, such as using collect() on large datasets.

• Data skew can cause uneven distribution of data across partitions, impacting processing time.

• Resource constraints like insufficient memory or CPU can result in slow Spark jobs.

• Improper configuration settings, su...read more
Answered by AI
Add your answer

Q1. What is Pyspark?
Ans. 
Pyspark is a Python API for Apache Spark, a powerful open-source distributed computing system.
• Pyspark is used for processing large datasets in parallel across a cluster of computers.

• It provides high-level APIs in Python for Spark programming.

• Pyspark allows seamless integration with other Python libraries like Pandas and NumPy.

• Example: Using Pyspark to perform data analysis and machine learning tasks on big data sets.
Answered by AI
Add your answer
Q2. What is Pyspark SQL?
Ans. 
Pyspark SQL is a module in Apache Spark that provides a SQL interface for working with structured data.
• Pyspark SQL allows users to run SQL queries on Spark dataframes.

• It provides a more concise and user-friendly way to interact with data compared to traditional Spark RDDs.

• Users can leverage the power of SQL for data manipulation and analysis within the Spark ecosystem.
Answered by AI
Add your answer
Q3. How to merge 2 dataframes of different schema?
Ans. 
To merge 2 dataframes of different schema, use join operations or data transformation techniques.
• Use join operations like inner join, outer join, left join, or right join based on the requirement.

• Perform data transformation to align the schemas before merging.

• Use tools like Apache Spark, Pandas, or SQL to merge dataframes with different schemas.
Answered by AI
Add your answer
Q4. What is Pyspark streaming?
Ans. 
Pyspark streaming is a scalable and fault-tolerant stream processing engine built on top of Apache Spark.
• Pyspark streaming allows for real-time processing of streaming data.

• It provides high-level APIs in Python for creating streaming applications.

• Pyspark streaming supports various data sources like Kafka, Flume, Kinesis, etc.

• It enables windowed computations and stateful processing for handling streaming data.

• Example: C...read more
Answered by AI
Add your answer

Q1. Spark Architecture
Add your answer
Q2. Explain the Spark architecture with example
Ans. 
Spark architecture includes driver, cluster manager, and worker nodes for distributed processing.
• Spark architecture consists of a driver program that manages the execution of tasks on worker nodes.

• Cluster manager is responsible for allocating resources and scheduling tasks across worker nodes.

• Worker nodes execute the tasks and store data in memory or disk for processing.

• Example: In a Spark application, the driver progr...read more
Answered by AI
Add your answer

Q1. Questions on SQL complex queries
Add your answer
Q2. Handling ADF pipelines
Ans. 
Handling ADF pipelines involves designing, building, and monitoring data pipelines in Azure Data Factory.
• Designing data pipelines using ADF UI or code

• Building pipelines with activities like copy data, data flow, and custom activities

• Monitoring pipeline runs and debugging issues

• Optimizing pipeline performance and scheduling triggers
Answered by AI
Add your answer
Q3. Schedules and triggers
Add your answer
Q4. About project work and complexity faced and handling issues
Add your answer

Q1. Salary discussion
Add your answer

Q1. Complete interview was on project discussion
Add your answer
Q2. On ensemble models some questions on stats
Add your answer

Q1. It wnt well basics of hadoop,spark and my project
Add your answer
Q2. Basics of sql and joins
Add your answer

Basics of sql and joins