Fanatics Data Engineer 3 Interview Questions and Answers

Q1. Resume screening Self introduction Roles and responsibilities
Add your answer

Given task Statics standard deviations Attrition Average of given table values and Given graph economi graph and poverty graph base on that need to gave answers 30 qustion and 60 min time duration

Q1. Versant test should be need to attend Hirepro will have interview prosess in YouTube will check and prepare mid level hard
Add your answer
Q2. We can goo for online prosess how it is
Add your answer

Q1. Manger Level round they will ask KPI NPS AHT standard deviations Service Level handling Root cause Analysis Weighted average Same product how will calculate Totall Swer % how will get Many of like bpi <20 ...read more
Add your answer

Q1. Salary discussion and update structure of our role and responsibilities
Add your answer

Q1. How would you create a pipeline for ADLS to SQL data movement?
Add your answer
Q2. How would you create a pipeline from REST API to ADLS? What is there are 8 million rows of records?
Add your answer
Q3. IF data needs filtering, joining and aggregation, how would you do it with ADF?
Add your answer
Q4. Explain medallion architecture.
Add your answer
Q5. Explain medallion with databricks
Add your answer

Q1. Basic questions and salary expectation.
Add your answer

Q1. Scenario based questions on Azure data factory and pipelines
Add your answer
Q2. Optimisation technic to improve the performance of databricks
Add your answer
Q3. What is Autoloader
Add your answer
Q4. What is unity catalog
Add your answer
Q5. How you do the alerting mechanism in adf for failed pipelines
Add your answer

Q1. Sql constrainsts, star schema, dml dcl commands
Add your answer
Q2. About cureent project and responsibilities
Add your answer

Q1. Current projects and resposibilities
Add your answer
Q2. Where vs having, reason for job change
Add your answer

Q1. How do you optimize SQL queries?
Ans. 
Optimizing SQL queries involves using indexes, avoiding unnecessary joins, and optimizing the query structure.
• Use indexes on columns frequently used in WHERE clauses

• Avoid using SELECT * and only retrieve necessary columns

• Optimize joins by using INNER JOIN instead of OUTER JOIN when possible

• Use EXPLAIN to analyze query performance and make necessary adjustments
Answered by AI
Add your answer
Q2. How do you do performance optimization in Spark. Tell how you did it in you project.
Ans. 
Performance optimization in Spark involves tuning configurations, optimizing code, and utilizing caching.
• Tune Spark configurations such as executor memory, number of executors, and shuffle partitions.

• Optimize code by reducing unnecessary shuffles, using efficient transformations, and avoiding unnecessary data movements.

• Utilize caching to store intermediate results in memory and avoid recomputation.

• Example: In my projec...read more
Answered by AI
Add your answer
Q3. What is SparkContext and SparkSession?
Ans. 
SparkContext is the main entry point for Spark functionality, while SparkSession is the entry point for Spark SQL.
• SparkContext is the entry point for low-level API functionality in Spark.

• SparkSession is the entry point for Spark SQL functionality.

• SparkContext is used to create RDDs (Resilient Distributed Datasets) in Spark.

• SparkSession provides a unified entry point for reading data from various sources and performing
Answered by AI
Add your answer
Q4. When a spark job is submitted, what happens at backend. Explain the flow.
Ans. 
When a spark job is submitted, various steps are executed at the backend to process the job.
• The job is submitted to the Spark driver program.

• The driver program communicates with the cluster manager to request resources.

• The cluster manager allocates resources (CPU, memory) to the job.

• The driver program creates DAG (Directed Acyclic Graph) of the job stages and tasks.

• Tasks are then scheduled and executed on worker nodes ...read more
Answered by AI
View 1 more answer
Q5. Calculate second highest salary using SQL as well as pyspark.
Ans. 
Calculate second highest salary using SQL and pyspark
• Use SQL query with ORDER BY and LIMIT to get the second highest salary

• In pyspark, use orderBy() and take() functions to achieve the same result
Answered by AI
Add your answer
Q6. 2 types of modes for Spark architecture ?
Ans. 
The two types of modes for Spark architecture are standalone mode and cluster mode.
• Standalone mode: Spark runs on a single machine with a single JVM and is suitable for development and testing.

• Cluster mode: Spark runs on a cluster of machines managed by a cluster manager like YARN or Mesos for production workloads.
Answered by AI
Add your answer
Q7. If you want very less latency - which is better standalone or client mode?
Ans. 
Client mode is better for very less latency due to direct communication with the cluster.
• Client mode allows direct communication with the cluster, reducing latency.

• Standalone mode requires an additional layer of communication, increasing latency.

• Client mode is preferred for real-time applications where low latency is crucial.
Answered by AI
Add your answer

Q1. Scenario based. Write SQL and pyspark code for a dataset.
Add your answer
Q2. If you have to find latest record based on latest timestamp in a table for a particular customer(table is having history) , how will you do it. Self join and nested query will be expensive. Optimized query...read more
Add your answer

Q1. Can you introduce yourself and describe your current project experience?
Ans. 
I am a Senior Data Engineer with experience in building scalable data pipelines and optimizing data processing workflows.
• Experience in designing and implementing ETL processes using tools like Apache Spark and Airflow

• Proficient in working with large datasets and optimizing query performance

• Strong background in data modeling and database design

• Worked on projects involving real-time data processing and streaming analytic
Answered by AI
Add your answer
Q2. Decorators in python
Ans. 
Decorators in Python are functions that modify the behavior of other functions or methods.
• Decorators are defined using the @decorator_name syntax before a function definition.

• They can be used to add functionality to existing functions without modifying their code.

• Decorators can be used for logging, timing, authentication, and more.

• Example: @staticmethod decorator in Python is used to define a static method in a class.
Answered by AI
Add your answer
Q3. What is the SQL query to group by employee ID in order to combine the first name and last name with a space?
Ans. 
SQL query to group by employee ID and combine first name and last name with a space
• Use the GROUP BY clause to group by employee ID

• Use the CONCAT function to combine first name and last name with a space

• Select employee ID, CONCAT(first_name, ' ', last_name) AS full_name
Answered by AI
Add your answer
Q4. What are constructors in Python?
Ans. 
Constructors in Python are special methods used for initializing objects. They are called automatically when a new instance of a class is created.
• Constructors are defined using the __init__() method in a class.

• They are used to initialize instance variables of a class.

• Example: class Person: def __init__(self, name, age): self.name = name self.age = age person1 = Person('Alice', 30)
Answered by AI
Add your answer
Q5. Indexing in sql
Ans. 
Indexing in SQL is a technique used to improve the performance of queries by creating a data structure that allows for faster retrieval of data.
• Indexes are created on columns in a database table to speed up the retrieval of rows that match a certain condition in a WHERE clause.

• Indexes can be created using CREATE INDEX statement in SQL.

• Types of indexes include clustered indexes, non-clustered indexes, unique indexes, an...read more
Answered by AI
Add your answer
Q6. Why spark works well with parquet files?
Ans. 
Spark works well with Parquet files due to its columnar storage format, efficient compression, and ability to push down filters.
• Parquet files are columnar storage format, which aligns well with Spark's processing model of working on columns rather than rows.

• Parquet files support efficient compression, reducing storage space and improving read performance in Spark.

• Spark can push down filters to Parquet files, allowing f...read more
Answered by AI
Add your answer

Q1. What are different type of joins available in Databricks?
Ans. 
Different types of joins available in Databricks include inner join, outer join, left join, right join, and cross join.
• Inner join: Returns only the rows that have matching values in both tables.

• Outer join: Returns all rows when there is a match in either table.

• Left join: Returns all rows from the left table and the matched rows from the right table.

• Right join: Returns all rows from the right table and the matched rows ...read more
Answered by AI
Add your answer
Q2. How do you make your data pipeline fault tolerant?
Ans. 
Implementing fault tolerance in a data pipeline involves redundancy, monitoring, and error handling.
• Use redundant components to ensure continuous data flow

• Implement monitoring tools to detect failures and bottlenecks

• Set up automated alerts for immediate response to issues

• Design error handling mechanisms to gracefully handle failures

• Use checkpoints and retries to ensure data integrity
Answered by AI
Add your answer
Q3. What is AutoLoader?
Ans. 
AutoLoader is a feature in data engineering that automatically loads data from various sources into a data warehouse or database.
• Automates the process of loading data from different sources

• Reduces manual effort and human error

• Can be scheduled to run at specific intervals

• Examples: Apache Nifi, AWS Glue
Answered by AI
Add your answer

Q1. How do you connect to different services in Azure?
Ans. 
To connect to different services in Azure, you can use Azure SDKs, REST APIs, Azure Portal, Azure CLI, and Azure PowerShell.
• Use Azure SDKs for programming languages like Python, Java, C#, etc.

• Utilize REST APIs to interact with Azure services programmatically.

• Access and manage services through the Azure Portal.

• Leverage Azure CLI for command-line interface interactions.

• Automate tasks using Azure PowerShell scripts.
Answered by AI
Add your answer
Q2. What are linked Services?
Ans. 
Linked Services are connections to external data sources or destinations in Azure Data Factory.
• Linked Services define the connection information needed to connect to external data sources or destinations.

• They can be used in Data Factory pipelines to read from or write to external systems.

• Examples of Linked Services include Azure Blob Storage, Azure SQL Database, and Amazon S3.
Answered by AI
Add your answer

Q1. Lets say you have table 1 with values 1,2,3,5,null,null,0 and table 2 has null,2,4,7,3,5 What would be the output after inner join?
Ans. 
The output after inner join of table 1 and table 2 will be 2,3,5.
• Inner join only includes rows that have matching values in both tables.

• Values 2, 3, and 5 are present in both tables, so they will be included in the output.

• Null values are not considered as matching values in inner join.
Answered by AI
Add your answer
Q2. Lets say you have customers table with customerID and customer name, Orders table with OrderId and CustomerID. write a query to find the customer name who placed the maximum orders. if more than one person...read more
Add your answer
Q3. Spark Architecture, Optimisation techniques
Add your answer
Q4. Some personal questions.
Add your answer

Q1. Explain the entire architecture of a recent project you are working on in your organisation.
Ans. 
The project involves building a data pipeline to ingest, process, and analyze large volumes of data from various sources in Azure.
• Utilizing Azure Data Factory for data ingestion and orchestration

• Implementing Azure Databricks for data processing and transformation

• Storing processed data in Azure Data Lake Storage

• Using Azure Synapse Analytics for data warehousing and analytics

• Leveraging Azure DevOps for CI/CD pipeline aut
Answered by AI
Add your answer
Q2. How do you design an effective ADF pipeline and what all metrics and considerations you should keep in mind while designing?
Ans. 
Designing an effective ADF pipeline involves considering various metrics and factors.
• Understand the data sources and destinations

• Identify the dependencies between activities

• Optimize data movement and processing for performance

• Monitor and track pipeline execution for troubleshooting

• Consider security and compliance requirements

• Use parameterization and dynamic content for flexibility

• Implement error handling and retries fo
Answered by AI
Add your answer
Q3. Lets say you have a very huge data volume and in terms of performance how would you slice and dice the data in such a way that you can boost the performance?
Add your answer
Q4. Lets say you have to reconstruct a table and we have to preserve the historical data ? ( i couldnt answer that but please refer to SCD)
Add your answer
Q5. We have adf and databricks both, i can achieve transformation , fetching the data and loading the dimension layer using adf also but why do we use databricks if both have the similar functionality for few ...read more
Add your answer

Q1. Basic HR questions
Add your answer