Cyril Amarchand Mangaldas Interview Questions and Answers

AWS services used include S3, Redshift, Glue, EMR, and Lambda in a scalable and cost-effective architecture.

• AWS S3 for storing large amounts of data

• AWS Redshift for data warehousing and analytics

• AWS Glue for ETL processes

• AWS EMR for big data processing

• AWS Lambda for serverless computing

Developed a real-time data processing pipeline for analyzing customer behavior

• Designed and implemented data ingestion process using Apache Kafka

• Utilized Apache Spark for data processing and analysis

• Built data models and visualizations using tools like Tableau

• Implemented machine learning algorithms for predictive analytics

Spark submit command is used to submit Spark applications to a cluster

• Used to launch Spark applications on a cluster

• Requires specifying the application JAR file, main class, and any arguments

• Can set various configurations like memory allocation, number of executors, etc.

• Example: spark-submit --class com.example.Main --master yarn --deploy-mode cluster myApp.jar arg1 arg2

Developed a real-time data processing pipeline for analyzing customer behavior

• Designed and implemented data ingestion process using Apache Kafka

• Utilized Apache Spark for data processing and analysis

• Built data models and visualizations using tools like Tableau

• Implemented machine learning algorithms for predictive analytics

To configure a cluster for 100 TB data, consider factors like storage capacity, processing power, network bandwidth, and fault tolerance.

• Choose a distributed storage system like HDFS or Amazon S3 for scalability and fault tolerance.

• Select high-capacity servers with sufficient RAM and CPU for processing large volumes of data.

• Ensure high-speed network connections between nodes to facilitate data transfer.

• Implement data r...read more

Our current project architecture involves a microservices-based approach with data pipelines for real-time processing.

• Utilizing microservices architecture for scalability and flexibility

• Implementing data pipelines for real-time processing of large volumes of data

• Leveraging cloud services such as AWS or Azure for infrastructure

• Using technologies like Apache Kafka for streaming data

• Ensuring data quality and reliability t

Use a SQL query with a subquery to find the 2nd most ordered item in a category.

• Use a subquery to rank items within each category based on the number of orders

• Select the item with rank 2 within each category

• Order the results by category and rank to get the 2nd most ordered item in each category