Tech Mahindra Associate Data Engineer Interview Questions and Answers

Optimizing SQL queries involves using indexes, avoiding unnecessary joins, and optimizing the query structure.

• Use indexes on columns frequently used in WHERE clauses

• Avoid using SELECT * and only retrieve necessary columns

• Optimize joins by using INNER JOIN instead of OUTER JOIN when possible

• Use EXPLAIN to analyze query performance and make necessary adjustments

Performance optimization in Spark involves tuning configurations, optimizing code, and utilizing caching.

• Tune Spark configurations such as executor memory, number of executors, and shuffle partitions.

• Optimize code by reducing unnecessary shuffles, using efficient transformations, and avoiding unnecessary data movements.

• Utilize caching to store intermediate results in memory and avoid recomputation.

• Example: In my projec...read more

SparkContext is the main entry point for Spark functionality, while SparkSession is the entry point for Spark SQL.

• SparkContext is the entry point for low-level API functionality in Spark.

• SparkSession is the entry point for Spark SQL functionality.

• SparkContext is used to create RDDs (Resilient Distributed Datasets) in Spark.

• SparkSession provides a unified entry point for reading data from various sources and performing

When a spark job is submitted, various steps are executed at the backend to process the job.

• The job is submitted to the Spark driver program.

• The driver program communicates with the cluster manager to request resources.

• The cluster manager allocates resources (CPU, memory) to the job.

• The driver program creates DAG (Directed Acyclic Graph) of the job stages and tasks.

• Tasks are then scheduled and executed on worker nodes ...read more

Calculate second highest salary using SQL and pyspark

• Use SQL query with ORDER BY and LIMIT to get the second highest salary

• In pyspark, use orderBy() and take() functions to achieve the same result

The two types of modes for Spark architecture are standalone mode and cluster mode.

• Standalone mode: Spark runs on a single machine with a single JVM and is suitable for development and testing.

• Cluster mode: Spark runs on a cluster of machines managed by a cluster manager like YARN or Mesos for production workloads.

Client mode is better for very less latency due to direct communication with the cluster.

• Client mode allows direct communication with the cluster, reducing latency.

• Standalone mode requires an additional layer of communication, increasing latency.

• Client mode is preferred for real-time applications where low latency is crucial.

I am a Senior Data Engineer with experience in building scalable data pipelines and optimizing data processing workflows.

• Experience in designing and implementing ETL processes using tools like Apache Spark and Airflow

• Proficient in working with large datasets and optimizing query performance

• Strong background in data modeling and database design

• Worked on projects involving real-time data processing and streaming analytic

Decorators in Python are functions that modify the behavior of other functions or methods.

• Decorators are defined using the @decorator_name syntax before a function definition.

• They can be used to add functionality to existing functions without modifying their code.

• Decorators can be used for logging, timing, authentication, and more.

• Example: @staticmethod decorator in Python is used to define a static method in a class.

SQL query to group by employee ID and combine first name and last name with a space

• Use the GROUP BY clause to group by employee ID

• Use the CONCAT function to combine first name and last name with a space

• Select employee ID, CONCAT(first_name, ' ', last_name) AS full_name

Constructors in Python are special methods used for initializing objects. They are called automatically when a new instance of a class is created.

• Constructors are defined using the __init__() method in a class.

• They are used to initialize instance variables of a class.

• Example: class Person: def __init__(self, name, age): self.name = name self.age = age person1 = Person('Alice', 30)

Indexing in SQL is a technique used to improve the performance of queries by creating a data structure that allows for faster retrieval of data.

• Indexes are created on columns in a database table to speed up the retrieval of rows that match a certain condition in a WHERE clause.

• Indexes can be created using CREATE INDEX statement in SQL.

• Types of indexes include clustered indexes, non-clustered indexes, unique indexes, an...read more

Spark works well with Parquet files due to its columnar storage format, efficient compression, and ability to push down filters.

• Parquet files are columnar storage format, which aligns well with Spark's processing model of working on columns rather than rows.

• Parquet files support efficient compression, reducing storage space and improving read performance in Spark.

• Spark can push down filters to Parquet files, allowing f...read more

Different types of joins available in Databricks include inner join, outer join, left join, right join, and cross join.

• Inner join: Returns only the rows that have matching values in both tables.

• Outer join: Returns all rows when there is a match in either table.

• Left join: Returns all rows from the left table and the matched rows from the right table.

• Right join: Returns all rows from the right table and the matched rows ...read more

Implementing fault tolerance in a data pipeline involves redundancy, monitoring, and error handling.

• Use redundant components to ensure continuous data flow

• Implement monitoring tools to detect failures and bottlenecks

• Set up automated alerts for immediate response to issues

• Design error handling mechanisms to gracefully handle failures

• Use checkpoints and retries to ensure data integrity

AutoLoader is a feature in data engineering that automatically loads data from various sources into a data warehouse or database.

• Automates the process of loading data from different sources

• Reduces manual effort and human error

• Can be scheduled to run at specific intervals

• Examples: Apache Nifi, AWS Glue

To connect to different services in Azure, you can use Azure SDKs, REST APIs, Azure Portal, Azure CLI, and Azure PowerShell.

• Use Azure SDKs for programming languages like Python, Java, C#, etc.

• Utilize REST APIs to interact with Azure services programmatically.

• Access and manage services through the Azure Portal.

• Leverage Azure CLI for command-line interface interactions.

• Automate tasks using Azure PowerShell scripts.

Linked Services are connections to external data sources or destinations in Azure Data Factory.

• Linked Services define the connection information needed to connect to external data sources or destinations.

• They can be used in Data Factory pipelines to read from or write to external systems.

• Examples of Linked Services include Azure Blob Storage, Azure SQL Database, and Amazon S3.

I have a background in data analysis with experience in using tools like Python, SQL, and Tableau.

• I have a degree in Statistics and have worked as a Data Analyst for 3 years.

• My daily activities include cleaning and analyzing data, creating visualizations, and presenting insights to stakeholders.

• I use Python for data manipulation and analysis, SQL for querying databases, and Tableau for creating interactive dashboards.

• I...read more

Advanced Excel and Power BI are tools used for data analysis and visualization in companies and for clients.

• Advanced Excel allows for complex data manipulation, analysis, and visualization using features like pivot tables, macros, and VBA programming.

• Power BI is a business analytics tool that provides interactive visualizations and business intelligence capabilities, connecting to various data sources.

• These tools are u...read more

I have extensive experience in using Advanced Excel and Power BI for data analysis projects.

• Created complex formulas and macros in Excel to automate data processing tasks

• Designed interactive dashboards in Power BI to visualize and analyze data trends

• Integrated data from multiple sources into Power BI for comprehensive analysis

• Used Power Query and Power Pivot in Excel to manipulate and analyze large datasets

• Provided dat...read more

Credit and operations concepts in relation to KYC procedures and client data privacy.

• Credit refers to the extension of money or resources to a client based on their financial history and ability to repay.

• Operations involve the day-to-day processes and procedures within a financial institution to ensure smooth functioning.

• KYC procedures are used to verify the identity of clients to prevent fraud and money laundering.

• Pri...read more

I am a Senior Data Engineer with experience in developing data pipelines and optimizing data storage for various projects.

• Developed data pipelines using Apache Spark for real-time data processing

• Optimized data storage using technologies like Hadoop and AWS S3

• Worked on a project to analyze customer behavior and improve marketing strategies

My day-to-day job in the project involved designing and implementing data pipelines, optimizing data workflows, and collaborating with cross-functional teams.

• Designing and implementing data pipelines to extract, transform, and load data from various sources

• Optimizing data workflows to improve efficiency and performance

• Collaborating with cross-functional teams including data scientists, analysts, and business stakeholde...read more

DAGs handle fault tolerance by rerunning failed tasks and maintaining task dependencies.

• DAGs rerun failed tasks automatically to ensure completion.

• DAGs maintain task dependencies to ensure proper sequencing.

• DAGs can be configured to retry failed tasks a certain number of times before marking them as failed.

Shuffling is the process of redistributing data across partitions in a distributed computing environment.

• Shuffling is necessary when data needs to be grouped or aggregated across different partitions.

• It can be handled efficiently by minimizing the amount of data being shuffled and optimizing the partitioning strategy.

• Techniques like partitioning, combiners, and reducers can help reduce the amount of shuffling in MapRed

Repartition increases or decreases the number of partitions in a DataFrame, while Coalesce only decreases the number of partitions.

• Repartition can increase or decrease the number of partitions in a DataFrame, leading to a shuffle of data across the cluster.

• Coalesce only decreases the number of partitions in a DataFrame without performing a full shuffle, making it more efficient than repartition.

• Repartition is typically...read more

Incremental data is handled by identifying new data since the last update and merging it with existing data.

• Identify new data since last update

• Merge new data with existing data

• Update data warehouse or database with incremental changes

SCD stands for Slowly Changing Dimension, a concept in data warehousing to track changes in data over time.

• SCD is used to maintain historical data in a data warehouse.

• There are three types of SCD - Type 1, Type 2, and Type 3.

• Type 1 SCD overwrites old data with new data.

• Type 2 SCD creates a new record for each change, preserving history.

• Type 3 SCD maintains both old and new values in the same record.

• SCD is important for...read more

Reverse a string using SQL and Python codes.

• In SQL, use the REVERSE function to reverse a string.

• In Python, use slicing with a step of -1 to reverse a string.

Use Spark and SQL to find the top 5 countries with the highest population.

• Use Spark to load the data and perform data processing.

• Use SQL queries to group by country and sum the population.

• Order the results in descending order and limit to top 5.

• Example: SELECT country, SUM(population) AS total_population FROM table_name GROUP BY country ORDER BY total_population DESC LIMIT 5

To find different records for different joins using two tables

• Use the SQL query to perform different joins like INNER JOIN, LEFT JOIN, RIGHT JOIN, and FULL JOIN

• Identify the key columns in both tables to join on

• Select the columns from both tables and use WHERE clause to filter out the different records

A catalyst optimizer is a query optimization tool used in Apache Spark to improve performance by generating an optimal query plan.

• Catalyst optimizer is a rule-based query optimization framework in Apache Spark.

• It leverages rules to transform the logical query plan into a more optimized physical plan.

• The optimizer applies various optimization techniques like predicate pushdown, constant folding, and join reordering.

• By o...read more

Used query optimization techniques to improve performance in database queries.

• Utilized indexing to speed up search queries.

• Implemented query caching to reduce redundant database calls.

• Optimized SQL queries by restructuring joins and subqueries.

• Utilized database partitioning to improve query performance.

• Used query profiling tools to identify and optimize slow queries.

Use the len() function to check the length of the data frame.

• Use len() function to get the number of rows in the data frame.

• If the length is 0, then the data frame is empty.

• Example: if len(df) == 0: print('Data frame is empty')

Cores and worker nodes are decided based on the workload requirements and scalability needs of the data processing system.

• Consider the size and complexity of the data being processed

• Evaluate the processing speed and memory requirements of the tasks

• Take into account the parallelism and concurrency needed for efficient data processing

• Monitor the system performance and adjust cores and worker nodes as needed

Enforcing schema ensures that data conforms to a predefined structure and rules.

• Ensures data integrity by validating incoming data against predefined schema

• Helps in maintaining consistency and accuracy of data

• Prevents data corruption and errors in data processing

• Can lead to rejection of data that does not adhere to the schema