Evernorth Health Services Senior Data Engineer Interview Questions and Answers

Q1. All questions based on databricks and some pyspark , python , SQL only.
Add your answer
Q2. Learn windows function implementation in databricks note book.
Add your answer

Q1. This round about your salary discussion around.
Add your answer

Q1. Sql query to write max salary
Ans. 
Use SQL query with MAX function to find the highest salary in a table.
• Use SELECT MAX(salary) FROM table_name;

• Make sure to replace 'salary' with the actual column name in the table.

• Ensure proper permissions to access the table.
Answered by AI
Add your answer
Q2. What is dense rank in sql
Ans. 
Dense rank in SQL assigns a unique rank to each distinct row in a result set, with no gaps between the ranks.
• Dense rank is used to assign a rank to each row in a result set without any gaps.

• It differs from regular rank in that it does not skip ranks if there are ties.

• For example, if two rows have the same value and are ranked 1st, the next row will be ranked 2nd, not 3rd.
Answered by AI
Add your answer

Q1. What is spark cluster
Ans. 
Spark cluster is a group of interconnected computers that work together to process large datasets using Apache Spark.
• Consists of a master node and multiple worker nodes

• Master node manages the distribution of tasks and resources

• Worker nodes execute the tasks in parallel

• Used for processing big data and running distributed computing jobs
Answered by AI
Add your answer
Q2. How hive works in hdfs
Ans. 
Hive is a data warehouse system built on top of Hadoop for querying and analyzing large datasets stored in HDFS.
• Hive translates SQL-like queries into MapReduce jobs to process data stored in HDFS

• It uses a metastore to store metadata about tables and partitions

• HiveQL is the query language used in Hive, similar to SQL

• Hive supports partitioning, bucketing, and indexing for optimizing queries
Answered by AI
Add your answer

Python and sql tasks

Q1. Project related questions
Add your answer
Q2. Coding questions on pyspark windows
Add your answer

Q1. Managerial discussions, mostly in and around the previous projects
Add your answer

Q1. Coding in pyspark
Ans. 
Pyspark is a Python library for big data processing using Spark framework.
• Pyspark is used for processing large datasets in parallel.

• It provides APIs for data manipulation, querying, and analysis.

• Example: Using pyspark to read a CSV file and perform data transformations.
Answered by AI
Add your answer
Q2. Databricks optimisation technique
Ans. 
Databricks optimisation techniques improve performance and efficiency of data processing on the Databricks platform.
• Use cluster sizing and autoscaling to optimize resource allocation based on workload

• Leverage Databricks Delta for optimized data storage and processing

• Utilize caching and persisting data to reduce computation time

• Optimize queries by using appropriate indexing and partitioning strategies
Answered by AI
Add your answer
Q3. Aqe details in databricks
Ans. 
Databricks is a unified data analytics platform that provides a collaborative environment for data engineers.
• Databricks is built on top of Apache Spark and provides a workspace for data engineering tasks.

• It allows for easy integration with various data sources and tools for data processing.

• Databricks provides features like notebooks, clusters, and libraries for efficient data engineering workflows.
Answered by AI
Add your answer

Spark Optimization, Transformation, DLT, DL, Data Governance
Python
SQL

Q1. All medium level theory and SQL, python/pyspark code.
Add your answer

Q1. More in detail about tools and technologies, project handling, agile, CICD, scenario based.
Add your answer

Q1. How to work with nested json using pyspark
Ans. 
Working with nested JSON using PySpark involves using the StructType and StructField classes to define the schema and then using the select function to access nested fields.
• Define the schema using StructType and StructField classes

• Use the select function to access nested fields

• Use dot notation to access nested fields, for example df.select('nested_field.sub_field')
Answered by AI
Add your answer
Q2. How to implement scd2 step by step
Ans. 
Implementing SCD2 involves tracking historical changes in data over time.
• Identify the business key that uniquely identifies each record

• Add effective start and end dates to track when the record was valid

• Insert new records with updated data and end date of '9999-12-31'

• Update end date of previous record when a change occurs
Answered by AI
Add your answer

Q1. Write a SQL query to select data from table 2 where data exists in table 1
Ans. 
Use a SQL query to select data from table 2 where data exists in table 1
• Use a JOIN statement to link the two tables based on a common column

• Specify the columns you want to select from table 2

• Use a WHERE clause to check for existence of data in table 1
Answered by AI
Add your answer
Q2. After performing joins how many records would be retrieved for inner, left, right and outer joins
Ans. 
The number of records retrieved after performing joins depends on the type of join - inner, left, right, or outer.
• Inner join retrieves only the matching records from both tables

• Left join retrieves all records from the left table and matching records from the right table

• Right join retrieves all records from the right table and matching records from the left table

• Outer join retrieves all records from both tables, filling
Answered by AI
Add your answer

Q1. About previous company and reason for leaving
Add your answer

Q1. How the files will be read in AWS GLUE
Ans. 
Files can be read in AWS Glue using Data Catalog, crawlers, and Glue ETL jobs.
• Use AWS Glue Data Catalog to store metadata information about the files.

• Crawlers can automatically infer the schema of the files and populate the Data Catalog.

• Glue ETL jobs can then be used to read the files from various sources like S3, RDS, etc.

• Supports various file formats like CSV, JSON, Parquet, etc.
Answered by AI
Add your answer

Q1. How to get duplicate records
Ans. 
Duplicate records can be identified using SQL queries by comparing columns and using aggregate functions.
• Use GROUP BY clause with COUNT() function to identify duplicate records

• Use HAVING clause to filter out records with count greater than 1

• Join the table with itself on specific columns to find duplicates
Answered by AI
Add your answer
Q2. How to get the second highest salary
Add your answer