Mphasis Pyspark Developer Interview Questions, Process, and Tips

Optimisation in PySpark involves improving performance and efficiency of Spark jobs.

• Use partitioning to distribute data evenly across nodes

• Avoid shuffling data between nodes as much as possible

• Use broadcast variables for small lookup tables

• Cache intermediate results to avoid recomputation

• Optimize transformations and actions for better performance

Null handling in Spark involves handling missing or null values in data processing.

• Use functions like coalesce, na.fill, na.drop to handle null values

• Consider using when and otherwise functions for conditional null handling

• Be cautious of potential null pointer exceptions when working with null values

Airflow operators are used to define the tasks to be executed in a workflow.

• Operators are classes that define the logic to execute a task in Airflow.

• There are various types of operators such as BashOperator, PythonOperator, and more.

• Operators can be customized to suit specific task requirements.

• Operators can be chained together to create complex workflows.

• Example: BashOperator executes a bash command, PythonOperator ru

SQL coding for calculating average rolling, lead, and lag functions.

• Use window functions like ROWS BETWEEN and ORDER BY for calculating rolling averages.

• Use LEAD and LAG functions to access data from previous or next rows.

• Example: SELECT col1, AVG(col2) OVER (ORDER BY col1 ROWS BETWEEN 3 PRECEDING AND CURRENT ROW) AS rolling_avg FROM table_name;

Use Apache Sqoop to move 10 tables from MySQL to Google Cloud Storage.

• Use Apache Sqoop to import data from MySQL to HDFS

• Use Google Cloud Storage connector for Hadoop to move data from HDFS to GCS

• Create a script to automate the process for all 10 tables