CGI Group Senior Data Engineer Interview Questions, Process, and Tips

Q1. Coding in pyspark
Ans. 
Pyspark is a Python library for big data processing using Spark framework.
• Pyspark is used for processing large datasets in parallel.

• It provides APIs for data manipulation, querying, and analysis.

• Example: Using pyspark to read a CSV file and perform data transformations.
Answered by AI
Add your answer
Q2. Databricks optimisation technique
Ans. 
Databricks optimisation techniques improve performance and efficiency of data processing on the Databricks platform.
• Use cluster sizing and autoscaling to optimize resource allocation based on workload

• Leverage Databricks Delta for optimized data storage and processing

• Utilize caching and persisting data to reduce computation time

• Optimize queries by using appropriate indexing and partitioning strategies
Answered by AI
Add your answer
Q3. Aqe details in databricks
Ans. 
Databricks is a unified data analytics platform that provides a collaborative environment for data engineers.
• Databricks is built on top of Apache Spark and provides a workspace for data engineering tasks.

• It allows for easy integration with various data sources and tools for data processing.

• Databricks provides features like notebooks, clusters, and libraries for efficient data engineering workflows.
Answered by AI
Add your answer

Q1. Adf,etl,python,adb
Add your answer

Q1. What is scd type 2?
Ans. 
SCD type 2 is a method used in data warehousing to track historical changes by creating a new record for each change.
• SCD type 2 stands for Slowly Changing Dimension type 2

• It involves creating a new record in the dimension table whenever there is a change in the data

• The old record is marked as inactive and the new record is marked as current

• It allows for historical tracking of changes in data over time

• Example: If a cust...read more
Answered by AI
Add your answer
Q2. Pyspark question read CSV from folder and add column in each csv file and write it to different location.
Add your answer

Q1. Explain about yourself.
Ans. 
I am a Senior Data Engineer with 5+ years of experience in designing and implementing data pipelines for large-scale projects.
• Experienced in ETL processes and data warehousing

• Proficient in programming languages like Python, SQL, and Java

• Skilled in working with big data technologies such as Hadoop, Spark, and Kafka

• Strong understanding of data modeling and database management

• Excellent problem-solving and communication sk
Answered by AI
Add your answer
Q2. Explain about your current project.
Ans. 
Developing a real-time data processing system for analyzing customer behavior on e-commerce platform.
• Utilizing Apache Kafka for real-time data streaming

• Implementing Spark for data processing and analysis

• Creating machine learning models for customer segmentation

• Integrating with Elasticsearch for data indexing and search functionality
Answered by AI
Add your answer
Q3. Big data concept
Add your answer
Q4. Spark related questions
Add your answer

Spark Optimization, Transformation, DLT, DL, Data Governance
Python
SQL

Q1. Tell about experience
Ans. 
I have over 5 years of experience in data engineering, working with large datasets and implementing data pipelines.
• Developed and maintained ETL processes to extract, transform, and load data from various sources

• Optimized database performance and implemented data quality checks

• Worked with cross-functional teams to design and implement data solutions

• Utilized tools such as Apache Spark, Hadoop, and SQL for data processing

• ...read more
Answered by AI
Add your answer
Q2. What would you do when you are given a new task to do
Ans. 
I would start by understanding the requirements, breaking down the task into smaller steps, researching if needed, and then creating a plan to execute the task efficiently.
• Understand the requirements of the task

• Break down the task into smaller steps

• Research if needed to gather necessary information

• Create a plan to execute the task efficiently

• Communicate with stakeholders for clarification or updates

• Regularly track prog
Answered by AI
Add your answer

Q1. How to work with nested json using pyspark
Ans. 
Working with nested JSON using PySpark involves using the StructType and StructField classes to define the schema and then using the select function to access nested fields.
• Define the schema using StructType and StructField classes

• Use the select function to access nested fields

• Use dot notation to access nested fields, for example df.select('nested_field.sub_field')
Answered by AI
Add your answer
Q2. How to implement scd2 step by step
Ans. 
Implementing SCD2 involves tracking historical changes in data over time.
• Identify the business key that uniquely identifies each record

• Add effective start and end dates to track when the record was valid

• Insert new records with updated data and end date of '9999-12-31'

• Update end date of previous record when a change occurs
Answered by AI
Add your answer

Q1. Write a SQL query to select data from table 2 where data exists in table 1
Ans. 
Use a SQL query to select data from table 2 where data exists in table 1
• Use a JOIN statement to link the two tables based on a common column

• Specify the columns you want to select from table 2

• Use a WHERE clause to check for existence of data in table 1
Answered by AI
Add your answer
Q2. After performing joins how many records would be retrieved for inner, left, right and outer joins
Ans. 
The number of records retrieved after performing joins depends on the type of join - inner, left, right, or outer.
• Inner join retrieves only the matching records from both tables

• Left join retrieves all records from the left table and matching records from the right table

• Right join retrieves all records from the right table and matching records from the left table

• Outer join retrieves all records from both tables, filling
Answered by AI
Add your answer

Q1. About previous company and reason for leaving
Add your answer