i
Coforge
Filter interviews by
I will use PySpark to handle 100 files of 100 GB size in an end-to-end pipeline.
Use PySpark to distribute processing across a cluster of machines
Read files in parallel using SparkContext and SparkSession
Apply transformations and actions to process the data efficiently
Utilize caching and persisting to optimize performance
Implement fault tolerance and recovery mechanisms
Use appropriate data storage solutions like HDFS or
I used batch processing by breaking down large data sets into smaller chunks for easier processing.
Implemented batch processing using tools like Apache Spark or Hadoop
Chose batch processing for its ability to handle large volumes of data efficiently
Split data into smaller batches to process sequentially for better resource management
Code to print top two highest numbers from an array
Sort the array in descending order
Print the first two elements of the sorted array
Top trending discussions
I applied via Naukri.com and was interviewed in Nov 2024. There were 2 interview rounds.
The Aptitude Test session accesses mathematical and logical reasoning abilities
Vlookup is a function in Excel used to search for a value in a table and return a corresponding value from another column.
Vlookup stands for 'Vertical Lookup'
It is commonly used in Excel to search for a value in the leftmost column of a table and return a value in the same row from a specified column
Syntax: =VLOOKUP(lookup_value, table_array, col_index_num, [range_lookup])
Example: =VLOOKUP(A2, B2:D10, 3, FALSE) - searc...
My day in my previous organization involved analyzing large datasets, creating reports, and presenting findings to stakeholders.
Reviewing and cleaning large datasets to ensure accuracy
Creating visualizations and reports to communicate insights
Collaborating with team members to identify trends and patterns
Presenting findings to stakeholders in meetings or presentations
I possess strong technical skills in data analysis, including proficiency in programming languages, statistical analysis, and data visualization tools.
Proficient in programming languages such as Python, R, SQL
Skilled in statistical analysis and data modeling techniques
Experience with data visualization tools like Tableau, Power BI
Knowledge of machine learning algorithms and techniques
A Pivot Table is a data summarization tool used in spreadsheet programs to analyze, summarize, and present data in a tabular format.
Pivot tables allow users to reorganize and summarize selected columns and rows of data to obtain desired insights.
Users can easily group and filter data, perform calculations, and create visualizations using pivot tables.
Pivot tables are commonly used in Excel and other spreadsheet program...
To find the highest-paid employee in each department, we need to group employees by department and then select the employee with the highest salary in each group.
Group employees by department
Find the employee with the highest salary in each group
Retrieve the employee's name, salary, and department name
I applied via Naukri.com and was interviewed in Oct 2024. There were 2 interview rounds.
Optimizing SQL queries involves using indexes, avoiding unnecessary joins, and optimizing the query structure.
Use indexes on columns frequently used in WHERE clauses
Avoid using SELECT * and only retrieve necessary columns
Optimize joins by using INNER JOIN instead of OUTER JOIN when possible
Use EXPLAIN to analyze query performance and make necessary adjustments
Performance optimization in Spark involves tuning configurations, optimizing code, and utilizing caching.
Tune Spark configurations such as executor memory, number of executors, and shuffle partitions.
Optimize code by reducing unnecessary shuffles, using efficient transformations, and avoiding unnecessary data movements.
Utilize caching to store intermediate results in memory and avoid recomputation.
Example: In my projec...
SparkContext is the main entry point for Spark functionality, while SparkSession is the entry point for Spark SQL.
SparkContext is the entry point for low-level API functionality in Spark.
SparkSession is the entry point for Spark SQL functionality.
SparkContext is used to create RDDs (Resilient Distributed Datasets) in Spark.
SparkSession provides a unified entry point for reading data from various sources and performing
When a spark job is submitted, various steps are executed at the backend to process the job.
The job is submitted to the Spark driver program.
The driver program communicates with the cluster manager to request resources.
The cluster manager allocates resources (CPU, memory) to the job.
The driver program creates DAG (Directed Acyclic Graph) of the job stages and tasks.
Tasks are then scheduled and executed on worker nodes ...
Calculate second highest salary using SQL and pyspark
Use SQL query with ORDER BY and LIMIT to get the second highest salary
In pyspark, use orderBy() and take() functions to achieve the same result
The two types of modes for Spark architecture are standalone mode and cluster mode.
Standalone mode: Spark runs on a single machine with a single JVM and is suitable for development and testing.
Cluster mode: Spark runs on a cluster of machines managed by a cluster manager like YARN or Mesos for production workloads.
Client mode is better for very less latency due to direct communication with the cluster.
Client mode allows direct communication with the cluster, reducing latency.
Standalone mode requires an additional layer of communication, increasing latency.
Client mode is preferred for real-time applications where low latency is crucial.
posted on 31 Dec 2024
Apache Spark architecture includes a cluster manager, worker nodes, and driver program.
Apache Spark architecture consists of a cluster manager, which allocates resources and schedules tasks.
Worker nodes execute tasks and store data in memory or disk.
Driver program coordinates tasks and communicates with the cluster manager.
Spark applications run as independent sets of processes on a cluster, coordinated by the SparkCon...
reduceBy is used to aggregate data based on key, while groupBy is used to group data based on key.
reduceBy is a transformation that combines the values of each key using an associative function and a neutral 'zero value'.
groupBy is a transformation that groups the data based on a key and returns a grouped data set.
reduceBy is more efficient for aggregating data as it reduces the data before shuffling, while groupBy shu...
RDD is a low-level abstraction representing a distributed collection of objects, while DataFrame is a higher-level abstraction representing a distributed collection of data organized into named columns.
RDD is more suitable for unstructured data and low-level transformations, while DataFrame is more suitable for structured data and high-level abstractions.
DataFrames provide optimizations like query optimization and code...
The different modes of execution in Apache Spark include local mode, standalone mode, YARN mode, and Mesos mode.
Local mode: Spark runs on a single machine with one executor.
Standalone mode: Spark runs on a cluster managed by a standalone cluster manager.
YARN mode: Spark runs on a Hadoop cluster using YARN as the resource manager.
Mesos mode: Spark runs on a Mesos cluster with Mesos as the resource manager.
I applied via AmbitionBox and was interviewed in Nov 2024. There were 4 interview rounds.
I utilize tools such as Excel, Python, SQL, and Tableau for data analysis.
Excel for basic data manipulation and visualization
Python for advanced data analysis and machine learning
SQL for querying databases
Tableau for creating interactive visualizations
Data analysis of code in the context of data analysis.
Coding logical question paper.
posted on 17 Jul 2024
I applied via Naukri.com and was interviewed in Aug 2024. There were 2 interview rounds.
I am a Senior Data Engineer with experience in developing data pipelines and optimizing data storage for various projects.
Developed data pipelines using Apache Spark for real-time data processing
Optimized data storage using technologies like Hadoop and AWS S3
Worked on a project to analyze customer behavior and improve marketing strategies
My day-to-day job in the project involved designing and implementing data pipelines, optimizing data workflows, and collaborating with cross-functional teams.
Designing and implementing data pipelines to extract, transform, and load data from various sources
Optimizing data workflows to improve efficiency and performance
Collaborating with cross-functional teams including data scientists, analysts, and business stakeholde...
DAGs handle fault tolerance by rerunning failed tasks and maintaining task dependencies.
DAGs rerun failed tasks automatically to ensure completion.
DAGs maintain task dependencies to ensure proper sequencing.
DAGs can be configured to retry failed tasks a certain number of times before marking them as failed.
Shuffling is the process of redistributing data across partitions in a distributed computing environment.
Shuffling is necessary when data needs to be grouped or aggregated across different partitions.
It can be handled efficiently by minimizing the amount of data being shuffled and optimizing the partitioning strategy.
Techniques like partitioning, combiners, and reducers can help reduce the amount of shuffling in MapRed
Repartition increases or decreases the number of partitions in a DataFrame, while Coalesce only decreases the number of partitions.
Repartition can increase or decrease the number of partitions in a DataFrame, leading to a shuffle of data across the cluster.
Coalesce only decreases the number of partitions in a DataFrame without performing a full shuffle, making it more efficient than repartition.
Repartition is typically...
Incremental data is handled by identifying new data since the last update and merging it with existing data.
Identify new data since last update
Merge new data with existing data
Update data warehouse or database with incremental changes
SCD stands for Slowly Changing Dimension, a concept in data warehousing to track changes in data over time.
SCD is used to maintain historical data in a data warehouse.
There are three types of SCD - Type 1, Type 2, and Type 3.
Type 1 SCD overwrites old data with new data.
Type 2 SCD creates a new record for each change, preserving history.
Type 3 SCD maintains both old and new values in the same record.
SCD is important for...
Reverse a string using SQL and Python codes.
In SQL, use the REVERSE function to reverse a string.
In Python, use slicing with a step of -1 to reverse a string.
Use Spark and SQL to find the top 5 countries with the highest population.
Use Spark to load the data and perform data processing.
Use SQL queries to group by country and sum the population.
Order the results in descending order and limit to top 5.
Example: SELECT country, SUM(population) AS total_population FROM table_name GROUP BY country ORDER BY total_population DESC LIMIT 5
To find different records for different joins using two tables
Use the SQL query to perform different joins like INNER JOIN, LEFT JOIN, RIGHT JOIN, and FULL JOIN
Identify the key columns in both tables to join on
Select the columns from both tables and use WHERE clause to filter out the different records
A catalyst optimizer is a query optimization tool used in Apache Spark to improve performance by generating an optimal query plan.
Catalyst optimizer is a rule-based query optimization framework in Apache Spark.
It leverages rules to transform the logical query plan into a more optimized physical plan.
The optimizer applies various optimization techniques like predicate pushdown, constant folding, and join reordering.
By o...
Used query optimization techniques to improve performance in database queries.
Utilized indexing to speed up search queries.
Implemented query caching to reduce redundant database calls.
Optimized SQL queries by restructuring joins and subqueries.
Utilized database partitioning to improve query performance.
Used query profiling tools to identify and optimize slow queries.
Use the len() function to check the length of the data frame.
Use len() function to get the number of rows in the data frame.
If the length is 0, then the data frame is empty.
Example: if len(df) == 0: print('Data frame is empty')
Cores and worker nodes are decided based on the workload requirements and scalability needs of the data processing system.
Consider the size and complexity of the data being processed
Evaluate the processing speed and memory requirements of the tasks
Take into account the parallelism and concurrency needed for efficient data processing
Monitor the system performance and adjust cores and worker nodes as needed
Enforcing schema ensures that data conforms to a predefined structure and rules.
Ensures data integrity by validating incoming data against predefined schema
Helps in maintaining consistency and accuracy of data
Prevents data corruption and errors in data processing
Can lead to rejection of data that does not adhere to the schema
I applied via Job Fair and was interviewed in Nov 2024. There were 2 interview rounds.
I was interviewed in Dec 2024.
I applied via Campus Placement and was interviewed in Dec 2024. There were 2 interview rounds.
Basics of mathematical ability and verbal ability
I applied via Company Website and was interviewed in Sep 2024. There were 2 interview rounds.
Platform - Hackerank
Duration - 2 Hours
Topics - Spark and SQL
Common file formats used in data storages include CSV, JSON, Parquet, Avro, and ORC. Parquet is best for compression.
CSV (Comma-Separated Values) - simple and widely used, but not efficient for large datasets
JSON (JavaScript Object Notation) - human-readable and easy to parse, but can be inefficient for storage
Parquet - columnar storage format that is highly efficient for compression and query performance
Avro - efficie...
Python program to find the most repeating substring in a list of words.
Iterate through each word in the list
Generate all possible substrings for each word
Count the occurrences of each substring using a dictionary
Find the substring with the highest count
Senior Software Engineer
4.8k
salaries
| ₹6.3 L/yr - ₹25.2 L/yr |
Technical Analyst
2.5k
salaries
| ₹9.3 L/yr - ₹38.4 L/yr |
Software Engineer
2k
salaries
| ₹2.2 L/yr - ₹9.5 L/yr |
Senior Test Engineer
1.8k
salaries
| ₹4.7 L/yr - ₹20.6 L/yr |
Technology Specialist
1.2k
salaries
| ₹11.8 L/yr - ₹42 L/yr |
Capgemini
Cognizant
Accenture
Infosys