Infosys Gcp Data Engineer Interview Questions and Answers

I have experience working on projects involving data processing, transformation, and analysis using GCP services like BigQuery, Dataflow, and Dataproc.

• Utilized BigQuery for storing and querying large datasets

• Implemented data pipelines using Dataflow for real-time data processing

• Utilized Dataproc for running Apache Spark and Hadoop clusters for data processing

• Worked on data ingestion and transformation using Cloud Stora

Google Cloud BigQuery is a fully-managed, serverless data warehouse that uses a distributed architecture for processing and analyzing large datasets.

• BigQuery uses a distributed storage system called Capacitor for storing and managing data.

• It uses a distributed query engine called Dremel for executing SQL-like queries on large datasets.

• BigQuery separates storage and compute, allowing users to scale compute resources ind...read more

List and tuple are both used to store collections of data, but they have some differences.

• Lists are mutable while tuples are immutable

• Lists use square brackets [] while tuples use parentheses ()

• Lists are typically used for collections of homogeneous data while tuples are used for heterogeneous data

• Lists have more built-in methods than tuples

GCP BigQuery is a serverless, highly scalable, and cost-effective data warehouse for analyzing big data sets.

• BigQuery is a fully managed, petabyte-scale data warehouse that enables super-fast SQL queries using the processing power of Google's infrastructure.

• BigQuery's architecture includes storage, Dremel execution engine, and SQL layer.

• Cloud Composer is a managed workflow orchestration service that helps you create, s...read more

select * from table limit 100 is faster

• Using 'select * from table' retrieves all rows from the table, which can be slower if the table is large

• Using 'select * from table limit 100' limits the number of rows retrieved, making it faster

• Limiting the number of rows fetched can improve query performance

SCD stands for Slowly Changing Dimension and Merge is a SQL operation used to update or insert data in BigQuery.

• SCD is used to track changes to data over time in a data warehouse

• Merge in BigQuery is used to perform insert, update, or delete operations in a single statement

• Example: MERGE INTO target_table USING source_table ON condition WHEN MATCHED THEN UPDATE SET col1 = value1 WHEN NOT MATCHED THEN INSERT (col1, col2)

BigQuery is a fully managed, serverless data warehouse that enables scalable analysis over petabytes of data.

• BigQuery uses a columnar storage format for efficient querying.

• It supports standard SQL for querying data.

• BigQuery allows for real-time data streaming for analysis.

• It integrates with various data sources like Google Cloud Storage, Google Sheets, etc.

• BigQuery provides automatic scaling and high availability.

Dataflow function to split sentence

• Use the Split transform in Dataflow to split the sentence into words

• Apply ParDo function to process each word individually

• Use regular expressions to handle punctuation and special characters

Window functions in BigQuery are used to perform calculations across a set of table rows related to the current row.

• Window functions allow you to perform calculations on a set of rows related to the current row

• They are used with the OVER() clause in SQL queries

• Common window functions include ROW_NUMBER(), RANK(), and NTILE()

• They can be used to calculate moving averages, cumulative sums, and more

Types of NoSQL databases in GCP include Firestore, Bigtable, and Datastore.

• Firestore is a flexible, scalable database for mobile, web, and server development.

• Bigtable is a high-performance NoSQL database service for large analytical and operational workloads.

• Datastore is a highly scalable NoSQL database for web and mobile applications.

Code to find max number of product by customer

• Iterate through each customer's purchases

• Keep track of the count of each product for each customer

• Find the product with the maximum count for each customer

Creating a dataframe in GCP Data Engineer

• Use the pandas library to create a dataframe

• Provide data in the form of a dictionary or list of lists

• Specify column names if needed

Bigquery is a cloud-based data warehousing tool used for analyzing large datasets quickly. Pubsub is a messaging service, Dataflow is a data processing tool, and Cloud Storage is a scalable object storage service.

• Bigquery is used for analyzing large datasets quickly

• Pubsub is a messaging service used for asynchronous communication between applications

• Dataflow is a data processing tool used for batch and stream processin...read more