EPAM Systems Data Engineer Interview Questions, Process, and Tips

Q1. Talk about your past experiences
Add your answer
Q2. Types of Variables in Scala
Ans. 
Scala has two types of variables - mutable and immutable.
• Scala has mutable variables that can be reassigned using the var keyword.

• Scala also has immutable variables that cannot be reassigned once they are initialized using the val keyword.

• Example: var mutableVariable = 10; val immutableVariable = 20;
Answered by AI
Add your answer
Q3. Explained in Detail about next Steps. Total 5 Rounds Including HR and HackerRank Test Round 1: HR Round 2: hackerrank assessment - If we clear this we move to Next Round Round 3: Technical Interview - Incl...read more
Add your answer

Hacker Rank Assessment - take home

Two coding questions on codility. One was easy and second medium. 10 MCQ questions on Big Data related technologies.

Q1. This round was scheduled for 1.5 hours and lasted 1 hrs 5 minutes. Discuss about projects done for previous company and architecture of the same.
Add your answer
Q2. Write code for printing duplicate numbers in a list.
Ans. 
Code to print duplicate numbers in a list.
• Iterate through the list and keep track of the count of each number using a dictionary.

• Print the numbers that have a count greater than 1.
Answered by AI
View 2 more answers
Q3. Scala traits, higher order functions, currying
Add your answer
Q4. Connecting Spark to Azure SQL Database.
Ans. 
Spark can connect to Azure SQL Database using JDBC driver.
• Download and install the JDBC driver for Azure SQL Database.

• Set up the connection string with the appropriate credentials.

• Use the JDBC API to connect Spark to Azure SQL Database.

• Example: val df = spark.read.jdbc(jdbcUrl, tableName, connectionProperties)

• Ensure that the firewall rules for the Azure SQL Database allow access from the Spark cluster.
Answered by AI
Add your answer
Q5. Elaboration of Spark optimization techniques. Types of transformations, shuffling.
Ans. 
Spark optimization techniques include partitioning, caching, and using appropriate transformations.
• Partitioning data can improve performance by reducing shuffling.

• Caching frequently used data can reduce the need for recomputation.

• Transformations like filter, map, and reduceByKey can be used to optimize data processing.

• Shuffling can be minimized by using operations like reduceByKey instead of groupByKey.

• Broadcasting sma...read more
Answered by AI
Add your answer
Q6. Difference between cache and persist, repartition and coalesce.
Ans. 
Cache and persist are used to store data in memory. Repartition and coalesce are used to change the number of partitions.
• Cache stores the data in memory for faster access while persist allows the user to choose the storage level.

• Repartition increases the number of partitions while coalesce decreases the number of partitions.

• Cache and persist are transformations while repartition and coalesce are actions.

• Cache and persi...read more
Answered by AI
Add your answer
Q7. Spark components and job execution steps.
Add your answer
Q8. Hive types of tables and difference between them
Ans. 
Hive has two types of tables - Managed and External. Managed tables are managed by Hive, while External tables are managed outside of Hive.
• Managed tables are created using 'CREATE TABLE' command and data is stored in Hive's warehouse directory

• External tables are created using 'CREATE EXTERNAL TABLE' command and data is stored outside of Hive's warehouse directory

• Managed tables are deleted when the table is dropped, whi...read more
Answered by AI
Add your answer

Q1. This was the final round of 1 hour and lasted 45 minutes.I was asked technical questions along with last companies project description.
Add your answer
Q2. Discuss project and it's architecture.
Ans. 
Developed a data pipeline to process and analyze customer behavior data.
• Used Apache Kafka for real-time data streaming

• Implemented data processing using Apache Spark

• Stored data in Hadoop Distributed File System (HDFS)

• Used Tableau for data visualization
Answered by AI
Add your answer
Q3. Write code to print reverse of a sentence word by word.
Ans. 
Code to print reverse of a sentence word by word.
• Split the sentence into words using space as delimiter

• Store the words in an array

• Print the words in reverse order
Answered by AI
Add your answer
Q4. Difference between RDD, Dataframe, Dataset.
Ans. 
RDD, Dataframe, and Dataset are data structures in Apache Spark with different characteristics and functionalities.
• RDD (Resilient Distributed Datasets) is a fundamental data structure in Spark that represents an immutable distributed collection of objects. It provides low-level APIs for distributed data processing and fault tolerance.

• Dataframe is a distributed collection of data organized into named columns. It is simi...read more
Answered by AI
Add your answer
Q5. Lineage graph, DAG formation, RDDs characteristics
Add your answer

Q1. Asked basic big data related questions. Hadoop, spark arch. Spark optimization, serialization. Hadoop datanode, namenode. SQL queries medium level.
Add your answer

Q1. How will you handle data skewness in spark
Ans. 
Data skewness can be handled in Spark by using techniques like partitioning, bucketing, and broadcasting.
• Partitioning the data based on a key column can distribute the data evenly across the cluster.

• Bucketing can further divide the data into smaller buckets based on a hash function.

• Broadcasting small tables can reduce the amount of data shuffled across the network.

• Using dynamic allocation can also help in handling data...read more
Answered by AI
Add your answer