EC-Mobility Intern Interview Questions and Answers

Optimisation techniques in Pyspark and sample code for SCD type 2

• Use broadcast variables to reduce data shuffling

• Partition data based on key columns to improve performance

• Use cache() or persist() to avoid recomputing data

• Use coalesce() or repartition() to reduce the number of partitions

• For SCD type 2, use merge() function to update or insert new records

• Example code for SCD type 2: https://github.com/awantik/pyspark-101

I have worked with huge data sets in the past.

• I have experience working with data sets ranging from a few gigabytes to several terabytes.

• One of the toughest problems I faced was optimizing a query that was taking too long to execute.

• I resolved it by analyzing the query plan and identifying the bottleneck, then optimizing the query accordingly.

• Another challenge I faced was dealing with data inconsistencies, which I reso...read more

Our project architecture follows a microservices approach with containerization using Docker and orchestration with Kubernetes.

• We have divided our application into smaller, independent services that communicate with each other through APIs.

• Each service is containerized using Docker, allowing for easy deployment and scaling.

• We use Kubernetes for orchestration, which automates the deployment, scaling, and management of o...read more

Streams in Java 11 provide a concise and efficient way to handle collections.

• Streams allow for functional-style operations on collections.

• They can be used to filter, map, and reduce data.

• Parallel streams can improve performance on large collections.

• Example: List names = Arrays.asList("John", "Jane", "Bob"); long count = names.stream().filter(name -> name.startsWith("J")).count();

• Example: List numbers = Arrays.asList(1,...read more