Publicis Sapient Data Engineer Interview Questions, Process, and Tips

Q1. How to Ensure Data loss in ETL pipeline
Add your answer
Q2. Why to spin up Dataproc cluster when there is serverless batch job exists.
Add your answer

Q1. What will happen if job has failed in pipeline and data processing cycle is over?
Ans. 
If a job fails in the pipeline and data processing cycle is over, it can lead to incomplete or inaccurate data.
• Incomplete data may affect downstream processes and analysis

• Data quality may be compromised if errors are not addressed

• Monitoring and alerting systems should be in place to detect and handle failures

• Re-running the failed job or implementing error handling mechanisms can help prevent issues in the future
Answered by AI
Add your answer
Q2. What is difference repartition and coalesce
Ans. 
Repartition increases the number of partitions in a DataFrame, while coalesce reduces the number of partitions without shuffling data.
• Repartition involves a full shuffle of the data across the cluster, which can be expensive.

• Coalesce minimizes data movement by only creating new partitions if necessary.

• Repartition is typically used when increasing parallelism or evenly distributing data, while coalesce is used for reduc...read more
Answered by AI
Add your answer

Q1. Write sql code to get the city1 city2 distance of table if city1 and city2 tables can repeat
Ans. 
SQL code to get the city1 city2 distance of table with repeating city1 and city2 values
• Use a self join on the table to match city1 and city2

• Calculate the distance between the cities using appropriate formula

• Consider using a subquery if needed
Answered by AI
Add your answer
Q2. How is data partitioned in pipeline
Ans. 
Data partitioning in a pipeline involves dividing data into smaller chunks for processing and analysis.
• Data can be partitioned based on a specific key or attribute, such as date, location, or customer ID.

• Partitioning helps distribute data processing tasks across multiple nodes or servers for parallel processing.

• Common partitioning techniques include range partitioning, hash partitioning, and list partitioning.

• Example: ...read more
Answered by AI
Add your answer

Q1. What Volume of data have you handled in your POCs ?
Ans. 
I have handled terabytes of data in my POCs, including data from various sources and formats.
• Handled terabytes of data in POCs

• Worked with data from various sources and formats

• Used tools like Hadoop, Spark, and SQL for data processing
Answered by AI
View 1 more answer
Q2. When will you decide to use repartition and coalesce?
Ans. 
Repartition is used for increasing partitions for parallelism, while coalesce is used for decreasing partitions to reduce shuffling.
• Repartition is used when there is a need for more partitions to increase parallelism.

• Coalesce is used when there are too many partitions and need to reduce them to avoid shuffling.

• Example: Repartition can be used before a join operation to evenly distribute data across partitions for bette...read more
Answered by AI
Add your answer
Q3. How will you design/configure a cluster if you have given 10 petabytes of data.
Ans. 
Designing/configuring a cluster for 10 petabytes of data involves considerations for storage capacity, processing power, network bandwidth, and fault tolerance.
• Consider using a distributed file system like HDFS or object storage like Amazon S3 to store and manage the large volume of data.

• Implement a scalable processing framework like Apache Spark or Hadoop to efficiently process and analyze the data in parallel.

• Utilize...read more
Answered by AI
Add your answer

Pyspark Coding Test - 2 Questions

Q1. Use of Vaccum in delta tables in terms of performance
Ans. 
Vaccum in delta tables helps improve performance by reclaiming space and optimizing file sizes.
• Vaccum operation helps optimize file sizes by removing small files and compacting larger files.

• It helps improve query performance by reducing the amount of data that needs to be scanned.

• Vaccum operation can be scheduled to run periodically to maintain optimal performance.

• It is recommended to run Vaccum on delta tables after m...read more
Answered by AI
Add your answer
Q2. SQL Coding Question on dense rank
Add your answer

Q1. Process one will use to solve a problem
Add your answer
Q2. How will consider inclusivity and diversity in workplace
Add your answer