IBM Gcp Data Engineer Interview Questions and Answers

Q1. Tools and Technology used in current project
Add your answer
Q2. What is Managed Table, external table, Materialised view
Add your answer
Q3. What is data flow? Working of data flow
Add your answer
Q4. Clusters in data proc, Types of clusters, Machine type used in cluster
Add your answer
Q5. Airflow - how to add email in airflow job - how to monitor jobs in airflow -Python Operator in airflow
Add your answer
Q6. Narrow , wide and broadcast transformation
Add your answer
Q7. Window functions
Add your answer
Q8. What is shuffle partition
Add your answer
Q9. How to identify file size in python
Add your answer
Q10. Which languages can be used in data flow
Add your answer

Q1. Explain Google cloud bigquery architecture?
Ans. 
Google Cloud BigQuery is a fully-managed, serverless data warehouse that uses a distributed architecture for processing and analyzing large datasets.
• BigQuery uses a distributed storage system called Capacitor for storing and managing data.

• It uses a distributed query engine called Dremel for executing SQL-like queries on large datasets.

• BigQuery separates storage and compute, allowing users to scale compute resources ind...read more
Answered by AI
View 1 more answer
Q2. Python: list and Tupple differences
Ans. 
List and tuple are both used to store collections of data, but they have some differences.
• Lists are mutable while tuples are immutable

• Lists use square brackets [] while tuples use parentheses ()

• Lists are typically used for collections of homogeneous data while tuples are used for heterogeneous data

• Lists have more built-in methods than tuples
Answered by AI
Add your answer

Q1. Questions on SQL Joins and Window Functions
Add your answer
Q2. GCP Big query and Cloud Storage qs
Add your answer

Q1. About overall IT experience
Add your answer
Q2. Project experience and services used
Ans. 
I have experience working on projects involving data processing, transformation, and analysis using GCP services like BigQuery, Dataflow, and Dataproc.
• Utilized BigQuery for storing and querying large datasets

• Implemented data pipelines using Dataflow for real-time data processing

• Utilized Dataproc for running Apache Spark and Hadoop clusters for data processing

• Worked on data ingestion and transformation using Cloud Stora
Answered by AI
Add your answer

Linked list using code for the given question

Q1. Which of these 2 select * from table and select * from table limit 100 is faster
Ans. 
select * from table limit 100 is faster
• Using 'select * from table' retrieves all rows from the table, which can be slower if the table is large

• Using 'select * from table limit 100' limits the number of rows retrieved, making it faster

• Limiting the number of rows fetched can improve query performance
Answered by AI
Add your answer
Q2. Explain scd and Merge in bigquery
Ans. 
SCD stands for Slowly Changing Dimension and Merge is a SQL operation used to update or insert data in BigQuery.
• SCD is used to track changes to data over time in a data warehouse

• Merge in BigQuery is used to perform insert, update, or delete operations in a single statement

• Example: MERGE INTO target_table USING source_table ON condition WHEN MATCHED THEN UPDATE SET col1 = value1 WHEN NOT MATCHED THEN INSERT (col1, col2)
Answered by AI
Add your answer
Q3. Architecture of bigquery
Ans. 
BigQuery is a fully managed, serverless data warehouse that enables scalable analysis over petabytes of data.
• BigQuery uses a columnar storage format for efficient querying.

• It supports standard SQL for querying data.

• BigQuery allows for real-time data streaming for analysis.

• It integrates with various data sources like Google Cloud Storage, Google Sheets, etc.

• BigQuery provides automatic scaling and high availability.
Answered by AI
Add your answer
Q4. Dataflow function to split sentence
Ans. 
Dataflow function to split sentence
• Use the Split transform in Dataflow to split the sentence into words

• Apply ParDo function to process each word individually

• Use regular expressions to handle punctuation and special characters
Answered by AI
Add your answer

Q1. Self intro and project architecture
Add your answer
Q2. What are the data sources used?
Ans. 
Various data sources such as databases, APIs, files, and streaming services are used for data ingestion and processing.
• Databases (e.g. MySQL, PostgreSQL)

• APIs (e.g. RESTful APIs)

• Files (e.g. CSV, JSON)

• Streaming services (e.g. Kafka, Pub/Sub)
Answered by AI
Add your answer
Q3. Bigquery architecture
Add your answer
Q4. Partition vs clustering
Ans. 
Partitioning is dividing data into smaller chunks for efficient storage and retrieval, while clustering is organizing data within those partitions based on a specific column.
• Partitioning is done at the storage level to distribute data across multiple nodes for better performance.

• Clustering is done at the query level to physically group data based on a specific column, improving query performance.

• Example: Partitioning b...read more
Answered by AI
Add your answer
Q5. Bq commands on create table and load csv file
Ans. 
Using bq commands to create a table and load a CSV file in Google BigQuery
• Use 'bq mk' command to create a new table in BigQuery

• Use 'bq load' command to load a CSV file into the created table

• Specify schema and source format when creating the table

• Specify source format and destination table when loading the CSV file

• Example: bq mk --table dataset.table_name schema.json

• Example: bq load --source_format=CSV dataset.table_nam
Answered by AI
Add your answer
Q6. Bq commands on show the schema of the table
Ans. 
Use 'bq show' command to display the schema of a table in BigQuery.
• Use 'bq show' command followed by the dataset and table name to display the schema.

• The schema includes the column names, data types, and mode (nullable or required).

• Example: bq show project_id:dataset.table_name
Answered by AI
Add your answer
Q7. Explain about leaf nodes and columnar storage.
Ans. 
Leaf nodes are the bottom nodes in a tree structure, while columnar storage stores data in columns rather than rows.
• Leaf nodes are the end nodes in a tree structure, containing actual data or pointers to data.

• Columnar storage stores data in columns rather than rows, allowing for faster query performance on specific columns.

• Columnar storage is commonly used in data warehouses and analytics databases.

• Leaf nodes are impor...read more
Answered by AI
Add your answer
Q8. How many slots are there in bigquery?
Ans. 
BigQuery does not have fixed slots, it dynamically allocates resources based on the query requirements.
• BigQuery does not have a fixed number of slots like traditional databases.

• It dynamically allocates resources based on the query requirements.

• The number of slots available for a query can vary depending on the complexity and size of the query.

• BigQuery's serverless architecture allows it to scale automatically to handle
Answered by AI
Add your answer

Need to write codes based on the scenarios

Q1. Reverser a number
Ans. 
Reverse a given number
• Convert the number to a string

• Reverse the string

• Convert the reversed string back to a number
Answered by AI
Add your answer
Q2. Recursion function for factorial
Ans. 
Recursion function to calculate factorial of a number
• Define a function that takes an integer as input

• Base case: if input is 0, return 1

• Recursive case: return input multiplied by factorial of input-1

• Example: factorial(5) = 5 * factorial(4) = 5 * 4 * factorial(3) = ... = 5 * 4 * 3 * 2 * 1 = 120
Answered by AI
Add your answer

Q1. Why you choose tcs
Ans. 
I chose TCS for its reputation, global presence, diverse opportunities, and focus on innovation.
• TCS is a renowned company with a strong reputation in the IT industry

• TCS has a global presence with offices in multiple countries, providing opportunities for international exposure

• TCS offers diverse opportunities for career growth and development in various domains

• TCS is known for its focus on innovation and cutting-edge te
Answered by AI
Add your answer
Q2. What is your notice period
Ans. 
I am currently on a 30-day notice period.
• My notice period is 30 days.

• I am required to work for 30 more days before leaving my current position.
Answered by AI
Add your answer

Q1. SQL: Find keys present in table A but not in B(B is old copy of A)
Ans. 
Use SQL to find keys present in table A but not in table B (old copy of A).
• Use a LEFT JOIN to combine tables A and B based on the key column

• Filter the results where the key column in table B is NULL

• This will give you the keys present in table A but not in table B
Answered by AI
Add your answer
Q2. SQL: 4th highest salary
Add your answer
Q3. Case Study: Using GCP's tool make a pipeline to transfer file from one GCS bucket to another
Ans. 
Use GCP Dataflow to transfer files between GCS buckets
• Create a Dataflow pipeline using Apache Beam to read from source bucket and write to destination bucket

• Use GCS connector to read and write files in Dataflow pipeline

• Set up appropriate permissions for Dataflow service account to access both buckets
Answered by AI
Add your answer
Q4. Case Study: A new joiner in IT, how will you explain flow of project and ownership of work. Considering my YOE 3 years
Add your answer
Q5. Explain your project, and reasons behind why did you choose airflow over other orchestration tool.
Add your answer
Q6. Discuss other orchestration tool in GCP
Ans. 
Cloud Composer is another orchestration tool in GCP
• Cloud Composer is a fully managed workflow orchestration service built on Apache Airflow

• It allows you to author, schedule, and monitor workflows that span across GCP services

• Cloud Composer provides a rich set of features like DAGs, plugins, and monitoring capabilities

• It integrates seamlessly with other GCP services like BigQuery, Dataflow, and Dataproc
Answered by AI
Add your answer