Infosys Data Engineer Interview Questions and Answers

Python dataframes are used to organize and manipulate data in a tabular format.

• Dataframes are created using the pandas library in Python.

• They allow for easy manipulation of data, such as filtering, sorting, and grouping.

• Dataframes can be used in various projects, such as data analysis, machine learning, and data visualization.

• Examples of using dataframes include analyzing sales data, predicting customer behavior, and visualizing stock market trends.

Developed a data pipeline to ingest, process, and analyze customer feedback data for product improvement.

• Designed and implemented ETL processes to extract data from various sources

• Utilized Apache Spark for data processing and analysis

• Built data visualizations to present insights to stakeholders

Spark has a master-slave architecture with a cluster manager and worker nodes.

• Spark has a driver program that communicates with a cluster manager to allocate resources and schedule tasks.

• The cluster manager can be standalone, Mesos, or YARN.

• Worker nodes execute tasks and store data in memory or on disk.

• Spark can also utilize external data sources like Hadoop Distributed File System (HDFS) or Amazon S3.

• Spark supports various APIs like SQL, Streaming, MLlib, and GraphX.

Clone tables inherit the privileges of the original table.

• Clone tables do inherit the privileges of the original table they were cloned from.

• Any user with privileges on the original table will also have the same privileges on the clone table.

• This can be useful for maintaining consistent access control across tables.

Examples of IaaS, PaaS, and SaaS include AWS (IaaS), Google App Engine (PaaS), and Salesforce (SaaS).

• IaaS - Infrastructure as a Service: AWS, Microsoft Azure, Google Cloud Platform

• PaaS - Platform as a Service: Google App Engine, Heroku, Microsoft Azure App Service

• SaaS - Software as a Service: Salesforce, Google Workspace, Microsoft Office 365

Some ADF activities include Copy Data, Execute Pipeline, Lookup, and Web Activity.

• Copy Data activity for moving data between sources and sinks

• Execute Pipeline activity for running another pipeline within a pipeline

• Lookup activity for retrieving data from a dataset

• Web Activity for calling a web service or API

Smb join is a method used to join two tables in SQL Server.

• Smb join stands for Sort Merge Bucket join.

• It is used when joining large tables.

• It involves sorting the tables and then merging them.

• It is an efficient join method for large tables with indexes.

• Example: SELECT * FROM table1 JOIN table2 ON table1.column = table2.column OPTION (HASH JOIN, MERGE JOIN, LOOP JOIN);

ADF stands for Azure Data Factory, a cloud-based data integration service. ADB stands for Azure Databricks, an Apache Spark-based analytics platform.

• ADF is used for data integration and orchestration, while ADB is used for big data analytics and machine learning.

• ADF provides a visual interface for building data pipelines, while ADB offers collaborative notebooks for data exploration and analysis.

• ADF supports various data sources and destinations, while ADB is optimized for pr...read more