Quantiphi Analytics Solutions Private Limited Associate Technical Architect Interview Questions, Process, and Tips

Azure storage BLOB is a general-purpose object storage solution, while ADLS Gen2 is optimized for big data analytics workloads.

• Azure storage BLOB is suitable for storing unstructured data like images, videos, and documents.

• ADLS Gen2 is designed for big data workloads and provides features like hierarchical namespace and optimized performance for analytics.

• Azure storage BLOB supports hot, cool, and archive storage tiers...read more

Reading files from different locations using Spark

• Use SparkSession to create a DataFrameReader

• Use the .option() method to specify the file location and format

• Use the .load() method to read the file into a DataFrame

Service principals are identities used by applications and services to access resources, while key vaults are secure storage for keys, secrets, and certificates.

• Service principals are used to authenticate and authorize applications and services to access resources in Azure

• Key vaults provide a secure and centralized location to store and manage cryptographic keys, secrets, and certificates

• Service principals can be grant...read more

The question covers various topics related to Spark, including repartitioning, coalescing, MLflow, partitioning of data, map, and map partition.

• Repartitioning and coalescing are used to change the number of partitions in an RDD/DataFrame.

• MLflow is an open-source platform for managing machine learning lifecycle.

• Partitioning of data is the process of dividing data into smaller, more manageable parts.

• Map and map partition...read more