ThoughtWorks Big Data Engineer Interview Questions and Answers

Q1. Self introduction
Add your answer

Maths grammar & communication

Q1. Typeing speed skills check same use paragraph
Add your answer
Q2. Typing speed 10 minutes
Add your answer

Q1. Move on next round easy
Add your answer

You're like this job opportunity

Q1. What are joins and its type
Add your answer
Q2. Tell me about Dax query
Add your answer

Q1. Write a few python queries and some live sql coding and pyspark coding
Add your answer
Q2. Machine learning questions. Way too deep. which is not even required
Add your answer

Q1. Sql regarding question was there
Add your answer

Basics of Statics and also verbal

Data structures and algorithms trees

Q1. Tell me about yourself
Add your answer
Q2. Previous experience questions
Add your answer

Q1. What is Pyspark?
Ans. 
Pyspark is a Python API for Apache Spark, a powerful open-source distributed computing system.
• Pyspark is used for processing large datasets in parallel across a cluster of computers.

• It provides high-level APIs in Python for Spark programming.

• Pyspark allows seamless integration with other Python libraries like Pandas and NumPy.

• Example: Using Pyspark to perform data analysis and machine learning tasks on big data sets.
Answered by AI
Add your answer
Q2. What is Pyspark SQL?
Ans. 
Pyspark SQL is a module in Apache Spark that provides a SQL interface for working with structured data.
• Pyspark SQL allows users to run SQL queries on Spark dataframes.

• It provides a more concise and user-friendly way to interact with data compared to traditional Spark RDDs.

• Users can leverage the power of SQL for data manipulation and analysis within the Spark ecosystem.
Answered by AI
Add your answer
Q3. How to merge 2 dataframes of different schema?
Ans. 
To merge 2 dataframes of different schema, use join operations or data transformation techniques.
• Use join operations like inner join, outer join, left join, or right join based on the requirement.

• Perform data transformation to align the schemas before merging.

• Use tools like Apache Spark, Pandas, or SQL to merge dataframes with different schemas.
Answered by AI
Add your answer
Q4. What is Pyspark streaming?
Ans. 
Pyspark streaming is a scalable and fault-tolerant stream processing engine built on top of Apache Spark.
• Pyspark streaming allows for real-time processing of streaming data.

• It provides high-level APIs in Python for creating streaming applications.

• Pyspark streaming supports various data sources like Kafka, Flume, Kinesis, etc.

• It enables windowed computations and stateful processing for handling streaming data.

• Example: C...read more
Answered by AI
Add your answer

Q1. SQL questions and partitioning
Add your answer
Q2. 3rd highest salary
Add your answer

Q1. Technical interview
Add your answer
Q2. SQL and data warehousing
Add your answer

Q1. Where do you see yourself in 5 years
Add your answer
Q2. Tell me about yourself
Add your answer