Publicis Sapient Senior Data Engineer Interview Questions, Process, and Tips

Q1. What is Application Master in Spark
Ans. 
Application Master in Spark is responsible for negotiating resources with the ResourceManager and executing tasks on the cluster.
• Responsible for negotiating resources with the ResourceManager

• Manages the execution of tasks on the cluster

• Monitors the progress of tasks and reports back to the driver program
Answered by AI
Add your answer
Q2. Architecture of Spark
Ans. 
Spark is a distributed computing framework that provides in-memory processing capabilities for big data analytics.
• Spark has a master-slave architecture with a central coordinator called the Spark Master and distributed workers called Spark Workers.

• It uses Resilient Distributed Datasets (RDDs) for fault-tolerant distributed data processing.

• Spark supports various data sources like HDFS, Cassandra, and S3 for input/output...read more
Answered by AI
Add your answer

Platform - Hackerank
Duration - 2 Hours
Topics - Spark and SQL

Q1. What are the common file formats used in data storages? Which one is best for compression?
Ans. 
Common file formats used in data storages include CSV, JSON, Parquet, Avro, and ORC. Parquet is best for compression.
• CSV (Comma-Separated Values) - simple and widely used, but not efficient for large datasets

• JSON (JavaScript Object Notation) - human-readable and easy to parse, but can be inefficient for storage

• Parquet - columnar storage format that is highly efficient for compression and query performance

• Avro - efficie...read more
Answered by AI
Add your answer
Q2. SQL Problem - Given the empoyee attendance table, write a query to print the employees who is abscent for more than cosecutive 10 days in their tenure.
Add your answer
Q3. Given the list of words, write the Python program to print the most repeating substring out of all words.
Ans. 
Python program to find the most repeating substring in a list of words.
• Iterate through each word in the list

• Generate all possible substrings for each word

• Count the occurrences of each substring using a dictionary

• Find the substring with the highest count
Answered by AI
Add your answer

Q1. Past projects in detail
Ans. 
Developed a real-time data processing system for a financial institution
• Designed and implemented data pipelines using Apache Kafka and Spark

• Optimized data processing workflows to handle large volumes of financial data

• Collaborated with cross-functional teams to ensure data quality and accuracy

• Implemented monitoring and alerting systems to detect anomalies in data processing

• Utilized machine learning models for fraud dete
Answered by AI
Add your answer
Q2. Questions on pyspark
Add your answer

Q1. Leadership base question
Add your answer
Q2. Situation base questions like what you will do if someone from your team is not performing well?
Add your answer

One question on pysaprk

Q1. Solve the query for the dataset
Add your answer
Q2. Solve it without using spark-sql
Add your answer