Infosys Data Engineer Interview Questions, Process, and Tips

Python dataframes are used to organize and manipulate data in a tabular format.

• Dataframes are created using the pandas library in Python.

• They allow for easy manipulation of data, such as filtering, sorting, and grouping.

• Dataframes can be used in various projects, such as data analysis, machine learning, and data visualization.

• Examples of using dataframes include analyzing sales data, predicting customer behavior, and v

Smb join is a method used to join two tables in SQL Server.

• Smb join stands for Sort Merge Bucket join.

• It is used when joining large tables.

• It involves sorting the tables and then merging them.

• It is an efficient join method for large tables with indexes.

• Example: SELECT * FROM table1 JOIN table2 ON table1.column = table2.column OPTION (HASH JOIN, MERGE JOIN, LOOP JOIN);

Spark has a master-slave architecture with a cluster manager and worker nodes.

• Spark has a driver program that communicates with a cluster manager to allocate resources and schedule tasks.

• The cluster manager can be standalone, Mesos, or YARN.

• Worker nodes execute tasks and store data in memory or on disk.

• Spark can also utilize external data sources like Hadoop Distributed File System (HDFS) or Amazon S3.

• Spark supports va...read more