EPAM Systems Data Engineer Interview Questions and Answers

Q1. Spark architecture, optimisations
Add your answer
Q2. Spark & Python Coding
Add your answer

Q1. Tell me about yourself
Add your answer
Q2. What tech stack are used
Ans. 
The tech stack used includes Python, SQL, Apache Spark, Hadoop, AWS, and Docker.
• Python for data processing and analysis

• SQL for database querying

• Apache Spark for big data processing

• Hadoop for distributed storage and processing

• AWS for cloud services

• Docker for containerization
Answered by AI
Add your answer

Q1. Talk about your past experiences
Add your answer
Q2. Types of Variables in Scala
Ans. 
Scala has two types of variables - mutable and immutable.
• Scala has mutable variables that can be reassigned using the var keyword.

• Scala also has immutable variables that cannot be reassigned once they are initialized using the val keyword.

• Example: var mutableVariable = 10; val immutableVariable = 20;
Answered by AI
Add your answer
Q3. Explained in Detail about next Steps. Total 5 Rounds Including HR and HackerRank Test Round 1: HR Round 2: hackerrank assessment - If we clear this we move to Next Round Round 3: Technical Interview - Incl...read more
Add your answer

Hacker Rank Assessment - take home

Q1. This round was scheduled for 1.5 hours and lasted 1 hrs 5 minutes. Discuss about projects done for previous company and architecture of the same.
Add your answer
Q2. Write code for printing duplicate numbers in a list.
Ans. 
This code identifies and prints duplicate numbers from a given list using a dictionary to track occurrences.
• Use a dictionary to count occurrences of each number.

• Iterate through the list and update the count in the dictionary.

• Print numbers that have a count greater than 1.

• Example: For the list [1, 2, 3, 2, 4, 3], the output should be 2 and 3.
Answered by AI
Add your answer
Q3. Scala traits, higher order functions, currying
Add your answer
Q4. Connecting Spark to Azure SQL Database.
Ans. 
Connecting Spark to Azure SQL Database involves configuring JDBC and using Spark's DataFrame API for data operations.
• Use the JDBC driver for Azure SQL Database to establish a connection.

• Example connection string: 'jdbc:sqlserver://<server>.database.windows.net:1433;database=<database>;user=<user>@<server>;password=<password>'

• Utilize Spark's DataFrame API to read and write data: df.write.jd...read more
Answered by AI
Add your answer
Q5. Elaboration of Spark optimization techniques. Types of transformations, shuffling.
Ans. 
Spark optimization techniques enhance performance through efficient data processing and resource management.
• Use DataFrames and Datasets for optimized execution plans.

• Leverage lazy evaluation to minimize unnecessary computations.

• Apply partitioning to distribute data evenly across nodes, e.g., using 'repartition' or 'coalesce'.

• Minimize shuffling by using narrow transformations like 'map' and 'filter' instead of wide tran...read more
Answered by AI
Add your answer
Q6. Difference between cache and persist, repartition and coalesce.
Ans. 
Cache stores data in memory for quick access, while persist saves it to disk. Repartition changes data distribution; coalesce reduces partitions.
• Cache: Stores DataFrame in memory for faster access during subsequent operations.

• Persist: Saves DataFrame to disk, allowing for fault tolerance but slower than cache.

• Repartition: Increases or decreases the number of partitions, potentially shuffling data across nodes.

• Coalesce:...read more
Answered by AI
Add your answer
Q7. Spark components and job execution steps.
Add your answer
Q8. Hive types of tables and difference between them
Ans. 
Hive supports two types of tables: Managed and External, each with distinct data management and storage characteristics.
• Managed Tables: Hive manages both the schema and the data. Dropping the table deletes the data.

• External Tables: Hive manages only the schema. Dropping the table does not delete the data, which remains in the external storage.

• Use Managed Tables for temporary data that can be recreated easily.

• Use Extern...read more
Answered by AI
Add your answer
Q9. This was the final round of 1 hour and lasted 45 minutes.I was asked technical questions along with last companies project description.
Add your answer
Q10. Discuss project and it's architecture.
Add your answer
Q11. Write code to print reverse of a sentence word by word.
Ans. 
This code reverses the order of words in a given sentence, providing a clear output of the reversed sentence.
• Split the Sentence: Use a method to split the sentence into an array of words. Example: 'Hello World' becomes ['Hello', 'World'].

• Reverse the Array: Utilize an array method to reverse the order of the words. Example: ['Hello', 'World'] becomes ['World', 'Hello'].

• Join the Words: Combine the reversed array back int...read more
Answered by AI
Add your answer
Q12. Difference between RDD, Dataframe, Dataset.
Ans. 
RDD, DataFrame, and Dataset are core abstractions in Apache Spark for handling distributed data processing.
• RDD (Resilient Distributed Dataset) is the fundamental data structure in Spark, representing an immutable distributed collection of objects.

• DataFrames are similar to RDDs but are optimized for performance and allow for schema-based operations, making them easier to use.

• Datasets combine the benefits of RDDs and Dat...read more
Answered by AI
Add your answer
Q13. Lineage graph, DAG formation, RDDs characteristics
Add your answer
Q14. Two coding questions on codility. One was easy and second medium. 10 MCQ questions on Big Data related technologies.
Add your answer

Q1. Asked basic big data related questions. Hadoop, spark arch. Spark optimization, serialization. Hadoop datanode, namenode. SQL queries medium level.
Add your answer

Q1. How will you handle data skewness in spark
Ans. 
Data skewness can be handled in Spark by using techniques like partitioning, bucketing, and broadcasting.
• Partitioning the data based on a key column can distribute the data evenly across the cluster.

• Bucketing can further divide the data into smaller buckets based on a hash function.

• Broadcasting small tables can reduce the amount of data shuffled across the network.

• Using dynamic allocation can also help in handling data...read more
Answered by AI
Add your answer