Interview Questions and Answers

Our project architecture follows a microservices approach with containerization using Docker and orchestration with Kubernetes.

• We have divided our application into smaller, independent services that communicate with each other through APIs.

• Each service is containerized using Docker, allowing for easy deployment and scaling.

• We use Kubernetes for orchestration, which automates the deployment, scaling, and management of our containers.

• Our services are written in different langua...read more

I have worked with huge data sets in the past.

• I have experience working with data sets ranging from a few gigabytes to several terabytes.

• One of the toughest problems I faced was optimizing a query that was taking too long to execute.

• I resolved it by analyzing the query plan and identifying the bottleneck, then optimizing the query accordingly.

• Another challenge I faced was dealing with data inconsistencies, which I resolved by implementing data validation and cleaning processe...read more

Use caching, minimize data transfer, optimize server-side code.

• Implement caching to reduce server requests

• Minimize data transfer by sending only necessary data

• Optimize server-side code for faster response time

Optimisation techniques in Pyspark and sample code for SCD type 2

• Use broadcast variables to reduce data shuffling

• Partition data based on key columns to improve performance

• Use cache() or persist() to avoid recomputing data

• Use coalesce() or repartition() to reduce the number of partitions

• For SCD type 2, use merge() function to update or insert new records

• Example code for SCD type 2: https://github.com/awantik/pyspark-101/blob/master/11-SCD2.ipynb

Yes, it is possible to connect to a database using JavaScript.

• JavaScript can connect to databases using APIs such as Web SQL, IndexedDB, and MongoDB.

• It is also possible to use Node.js to connect to databases using JavaScript.

• However, it is not recommended to connect to a database directly from client-side JavaScript due to security concerns.