100+ SAK Informatics Interview Questions and Answers

Coalesce is used to reduce the number of partitions in a DataFrame, while repartition is used to increase the number of partitions.

• Coalesce is a narrow transformation that can only decrease the number of partitions without shuffling the data.

• Repartition is a wide transformation that can both increase or decrease the number of partitions and involves shuffling the data across the cluster.

• Coalesce is more efficient for reducing the number of partitions when there is no need for...read more

Different file formats used in data engineering

• Common file formats include CSV, JSON, Parquet, Avro, ORC

• CSV (Comma-Separated Values) is a plain text format with data separated by commas

• JSON (JavaScript Object Notation) is a lightweight data interchange format

• Parquet is a columnar storage file format optimized for big data processing

• Avro is a row-oriented binary format with schema support

• ORC (Optimized Row Columnar) is a columnar storage file format for Hadoop

The release process involves planning, testing, and deploying software updates to users.

• Planning: Define scope, set deadlines, assign tasks

• Testing: Conduct regression testing, performance testing, etc.

• Deployment: Roll out updates to users in stages

• Feedback: Gather feedback from users for further improvements