Dataflow vs Dataproc, layering processing and curated environments in gcp , Data cleaning

Question

Accepted Answer

Dataflow and Dataproc are both processing services in GCP, but with different approaches and use cases.

Dataflow is a fully managed service for executing batch and streaming data processing pipelines.
Dataproc is a managed Spark and Hadoop service for running big data processing and analytics workloads.
Dataflow provides a serverless and auto-scaling environment, while Dataproc offers more control and flexibility.
Dataflow is suitable for real-time streaming and complex data transformations, while Dataproc is ideal for large-scale batch processing and machine learning.
Layering processing refers to the concept of applying multiple processing steps or transformations on data in a pipeline.
Curated environments in GCP refer to creating controlled and standardized environments for data processing and analysis.
Data cleaning involves the process of identifying and correcting or removing errors, inconsistencies, and inaccuracies in datasets.

Top EPAM Systems Senior Data Engineer interview questions & answers