Wipro Data Engineer Interview Questions, Process, and Tips

Q1. Joins, Aggregations, Projects detailing
Add your answer
Q2. Python and pyspark codes
Add your answer

Q1. Project question like how connected to s3 bucket through pyspark,HDFS basic commands likhe copy etc ,how and where created hive table (if used in your project).
Add your answer
Q2. Share variables in pyspark like broadcast and accumulator..(i will suggest go through pyspark official documentation once)
Add your answer
Q3. Sql joins- , how to read file in pyspark, job scheduling means resource allocation within and across spark..(ans- can go through job scheduling of pyspark documentaion )
Add your answer

Q1. What is pyspark
Ans. 
PySpark is a Python API for Apache Spark, a powerful open-source distributed computing system.
• PySpark is used for processing large datasets with distributed computing.

• It provides high-level APIs in Python for Spark programming.

• PySpark allows seamless integration with Python libraries like Pandas and NumPy.

• Example: PySpark can be used for data processing, machine learning, and real-time analytics.
Answered by AI
Add your answer
Q2. Difference between external and internal table
Ans. 
External tables reference data stored outside the database, while internal tables store data within the database.
• External tables are defined on data that is stored outside the database, such as in HDFS or S3.

• Internal tables store data within the database itself, typically in a managed storage like HDFS or S3.

• External tables do not delete data when dropped, while internal tables do.

• Internal tables are managed by the dat...read more
Answered by AI
Add your answer

Q1. Few Questions on Spark
Add your answer
Q2. Basic of Scala questions and coding
Add your answer

Quant , reasoning, english, coding

Q1. Technical concepts mentioned in resume
Add your answer
Q2. Some behavioural questions
Add your answer

Q1. Spark architecture
Add your answer
Q2. Spark Optimisation techniques
Ans. 
Spark optimization techniques aim to improve performance and efficiency of Spark jobs.
• Use partitioning to distribute data evenly

• Cache intermediate results to avoid recomputation

• Optimize shuffle operations by reducing data shuffling

• Use broadcast variables for small lookup tables

• Tune memory and executor settings for better performance
Answered by AI
Add your answer