DXC Technology Data Engineer Interview Questions, Process, and Tips

Design an Azure Data Factory pipeline to copy data with different delimiters.

• Use a Copy Data activity in Azure Data Factory to copy data from source folder to destination folder.

• Create a dataset for the source folder with multiple file formats to handle different delimiters.

• Use a mapping data flow to transform the data if needed before copying to the destination folder.

A pyspark program to read multiple csv files and create a data frame with count of records against each file.

• Use SparkSession to create a Spark application

• Read multiple csv files using spark.read.csv() method

• Use groupBy() and count() functions to get count of records for each file

• Create a new column to store the file name

• Join all data frames to create a final data frame with count of records against each file

Databricks workflows are a set of tasks and dependencies that are executed in a specific order to achieve a desired outcome.

• Databricks workflows are used to automate and orchestrate data engineering tasks.

• They define the sequence of steps and dependencies between tasks.

• Tasks can include data ingestion, transformation, analysis, and model training.

• Workflows can be scheduled to run at specific times or triggered by event...read more

The existing project design is a data pipeline for processing and analyzing customer data.

• The project uses Apache Kafka for real-time data ingestion.

• Data is stored in a distributed file system like Hadoop HDFS.

• Apache Spark is used for data processing and transformation.

• The processed data is loaded into a data warehouse like Amazon Redshift.

• The project includes monitoring and alerting mechanisms for data quality and pip