Wipro Data Engineer Interview Questions, Process, and Tips

Q1. How to migrate from Hive to Bigquery
Ans. 
Migrating from Hive to Bigquery involves exporting data from Hive, transforming it into a compatible format, and importing it into Bigquery.
• Export data from Hive using tools like Sqoop or Apache NiFi

• Transform the data into a compatible format like Avro or Parquet

• Import the transformed data into Bigquery using tools like Dataflow or Bigquery Data Transfer Service
Answered by AI
Add your answer

Q1. Pyspark - How to add new column to the data How to read data from Csv file
Ans. 
To add a new column to data in Pyspark, use 'withColumn' method. To read data from a CSV file, use 'spark.read.csv' method.
• To add a new column to data in Pyspark, use 'withColumn' method

• Example: df.withColumn('new_column', df['existing_column'] * 2)

• To read data from a CSV file, use 'spark.read.csv' method

• Example: df = spark.read.csv('file.csv', header=True, inferSchema=True)
Answered by AI
Add your answer

Q1. What's the use of broadcast and accumalator in spark
Ans. 
Broadcast and accumulator are used in Spark for efficient data sharing and aggregation across tasks.
• Broadcast variables are used to efficiently distribute large read-only data to all tasks in a Spark job.

• Accumulators are used for aggregating values from all tasks in a Spark job to a shared variable.

• Broadcast variables help in reducing data transfer costs and improving performance.

• Accumulators are used for tasks like co...read more
Answered by AI
Add your answer
Q2. Coalesce and repartition in spark
Ans. 
Coalesce and repartition are operations in Spark used to control the number of partitions in a DataFrame.
• Coalesce reduces the number of partitions without shuffling data, while repartition reshuffles data to create a specified number of partitions.

• Coalesce is more efficient when reducing partitions, as it minimizes data movement.

• Repartition is useful for evenly distributing data across a specified number of partitions.

• ...read more
Answered by AI
Add your answer

Q1. Pandas v/s spark
Add your answer
Q2. Architecture of spark
Ans. 
Spark is a distributed computing framework that provides in-memory processing capabilities for big data analytics.
• Spark has a master-slave architecture with a central coordinator called the Spark Master and distributed workers called Spark Workers.

• It uses Resilient Distributed Datasets (RDDs) for fault-tolerant distributed data processing.

• Spark supports various programming languages like Scala, Java, Python, and R for ...read more
Answered by AI
Add your answer

Q1. Write pyhton code for palindrome
Ans. 
Python code to check if a string is a palindrome or not.
• Define a function that takes a string as input.

• Use string slicing to reverse the input string.

• Compare the reversed string with the original string to check for palindrome.

• Return True if the string is a palindrome, False otherwise.
Answered by AI
Add your answer

2 coding questions asked moderate type

Q1. Questions on Oops python SQL asked
Add your answer

Q1. Hr discussed about goals and expectations.
Add your answer