Innoplexus Data Engineer Interview Questions and Answers

Q1. How to decide upon Spark cluster sizing?
Ans. 
Spark cluster sizing depends on workload, data size, memory requirements, and processing speed.
• Consider the size of the data being processed

• Take into account the memory requirements of the Spark jobs

• Factor in the processing speed needed for the workload

• Scale the cluster based on the number of nodes and cores required

• Monitor performance and adjust cluster size as needed
Answered by AI
Add your answer
Q2. Question on Spark internals
Add your answer

Implement a pipeline based on given conditions and data requirement

Q1. There were no questions as such. They want me join on very next day when the HR discussion was over and started itimidating me with aggressive approach even without discussing the salary part and the benef...read more
Add your answer

SQL test has been conducted nearer to 12-15 questions.

Q1. Normal discussion, about salary negotiation
Add your answer
Q2. Asked about my past company experience
Add your answer

Normal Aptitude Test

Python or SQL coding round

Q1. Components of Databricks
Ans. 
Databricks is a unified data analytics platform that includes components like Databricks Workspace, Databricks Runtime, and Databricks Delta.
• Databricks Workspace: Collaborative environment for data science and engineering teams.

• Databricks Runtime: Optimized Apache Spark cluster for data processing.

• Databricks Delta: Unified data management system for data lakes.
Answered by AI
Add your answer
Q2. How to read a json file
Ans. 
To read a JSON file, use a programming language's built-in functions or libraries to parse the file and extract the data.
• Use a programming language like Python, Java, or JavaScript to read the JSON file.

• Import libraries like json in Python or json-simple in Java to parse the JSON data.

• Use functions like json.load() in Python to load the JSON file and convert it into a dictionary or object.

• Access the data in the JSON fi...read more
Answered by AI
Add your answer
Q3. Second highest salary SQL
Ans. 
To find the second highest salary in SQL, use the MAX function with a subquery or the LIMIT clause.
• Use the MAX function with a subquery to find the highest salary first, then use a WHERE clause to exclude it and find the second highest salary.

• Alternatively, use the LIMIT clause to select the second highest salary directly.

• Make sure to handle cases where there may be ties for the highest salary.
Answered by AI
Add your answer
Q4. How to configure the spark while creating the cluster
Ans. 
Spark cluster configuration involves setting up memory, cores, and other parameters for optimal performance.
• Specify the number of executors and executor memory

• Set the number of cores per executor

• Adjust the driver memory based on the application requirements

• Configure shuffle partitions for efficient data processing

• Enable dynamic allocation for better resource utilization
Answered by AI
Add your answer

8 python coding
With 2 aptitude

Game based assessment.
More than 8 games need to play

Q1. Questions from python and also from SQL. SQL is must to place LatentView. Some projects related questions
Add your answer
Q2. How many joins are in SQL what is lamda function in python Difference between tuples and list
Ans. 
Answering questions related to SQL joins, lambda function in Python, and difference between tuples and lists.
• There are four types of joins in SQL: inner join, left join, right join, and full outer join.

• Lambda function is an anonymous function in Python that can take any number of arguments, but can only have one expression.

• Tuples are immutable and ordered collections of elements, while lists are mutable and ordered col...read more
Answered by AI
Add your answer