Spring ? SPRING BOOT? WHY SPRING BOOT? Use of Kafka? How Kafka can be used differently, give some use cases example? Spark ? And why we use spark, how it is different from Hadoop? Why we use spark if we can do same thing with Kafka also.? Java stream api ?

Question

Accepted Answer

Spring Boot is a framework for building Java applications. Kafka is a distributed streaming platform. Spark is a fast and general-purpose cluster computing system. Java Stream API is used for processing collections of data.

Spring Boot is preferred for its simplicity and convention-over-configuration approach.
Kafka is used for building real-time streaming data pipelines and applications.
Kafka can be used for various use cases like log aggregation, event sourcing, and messaging.
Spark is used for big data processing and analytics, providing faster processing than Hadoop.
Spark is different from Kafka as it focuses on batch processing and iterative algorithms.
Java Stream API provides functional programming capabilities for processing collections of data.
Java Stream API simplifies code and allows for parallel processing of data.

Accepted Answer

Spring is a framework used for IOC and DI.

Basically Spring boot is also a framework which provides auto configuration and other benefits. (Google for detailed answer)

Why Spring boot? Because it provides a feature called auto configuration which reduces a lot of configuration in your application , you can simply do all that in yaml or property file. As configuration is most difficult part for the programmer if he/she not using Google ;)

Kafka can be used in many ways. It can be used as logging system where you have multiple system and integrated logs in a huge quantity. Kafka can be used as data streaming ....for processing the data in Kafka streams. Kafka can also be used as a messaging queue. Kafka can also be used between two system as a bridge to share info, data or error mechanism. It just depend on where you want to use it. It can also be used in real time processing of data with spark and without spark.

If you are a developer and somehow got the opportunity of work in big data kind of project. So you must be aware about apache spark. Spark is used for real time processing. There are lot of system where spark is used. For example stock market where prices are changed continuosly at each second. Temperature data of whole world. Tesla uses spark for processing their cars data which is connected to IOT. Spark enables your system to collect as much data as you want and process it in real time. With Hadoop you are restricted to do that because with Hadoop you will execute the task in batch. That can be programmed by you .... Whether you want to processed a certain amount of data or process data at certain times. Just keep in mind both are great. Never tell interviewer that spark is best. It always depends on use cases. Where live data is important we generally use real time processing like spark. But if use case suggest that you need to execute tasks in batch to share Cpu time and performance. You had to use Hadoop and similar system.

With Kafka also you can do the same thing as with spark. But difference is how you use and where you use it. Kafka is great in read and write operation as it was designed for logs management. So, In my personal opinion you should always use Kafka with spark...and use benefits of both where necessary. This can greatly be explained by your team architect on how and where to use it.

+ Java stream api is used to processing the collection of data in stream itself. It saves a lot of cpu time as with stream data will not be convert to human readable format or java object format after each operation. Also it can be used as real time processing but doesn't have similar capabilities as spark. With spark and Kafka you can handle millions of record per second but you will not attain the same with Java stream api for so long. you will be able to but your program somehow fail at a certain point in a long go. Which is why I always opt for Kafka and spark. Kafka for read and write + holding capabilities of data. Spark for processing the data.