Smart Data Prefetching Using KNN to Improve Hadoop Performance

doi:10.21203/rs.3.rs-3221401/v1

Download PDF

Research Article

Smart Data Prefetching Using KNN to Improve Hadoop Performance

https://doi.org/10.21203/rs.3.rs-3221401/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

Hadoop is an open-source framework that enables the parallel processing of large data sets across a cluster of machines. It faces several challenges that can lead to poor performance, such as I/O operations, network data transmission, and high data access time. In recent years, researchers have explored prefetching techniques to reduce the data access time as a potential solution to these problems. Nevertheless, several issues must be considered to optimize the prefetching mechanism. These include launching the prefetch at an appropriate time to avoid conflicts with other operations and minimize waiting time, determining the amount of prefetched data to avoid overload and underload, and placing the prefetched data in a location that can be accessed efficiently when required. In this paper, we propose a smart prefetch mechanism that consists of three phases designed to address these issues. First, we enhance the task progress rate to calculate the optimal time for triggering prefetch operations. Next, we utilize K-Nearest Neighbor (KNN) clustering to identify which data blocks should be prefetched in each round, employing the data locality feature to determine the placement of prefetched data. Our experimental results demonstrate that our proposed smart prefetch mechanism improves job execution time by an average of 28.33% by increasing the rate of local tasks.

Hadoop performance

Prefetch technique

KNN

Data locality

Hadoop [1] enables the storage and analysis of BigData by incorporating two main components. The first, MapReduce [2], is a parallel programming model for processing a large amount of data through a cluster of machines in a distributed environment. The second component is the Hadoop Distributed File System (HDFS) [3] to hold a large volume of data. Although Hadoop brings some benefits it suffers from problems such as access latency in reading data from HDFS and increased input data transmission time from a remote node to a processing node. These issues can lead to prolonged runtime. Prefetching has been proposed as an efficient way to accelerate execution time by eliminating these delays. We could classify a prefetching strategy into two groups based on the source of fetching: HDFS prefetching to improve data access time from HDFS and Node prefetching to reduce data transmission time through the network. For this purpose, a prefetching thread is created to fetch data into the cache before they are requested. HDFS prefetch faces some challenges. Due to the master/slave architecture of HDFS, prefetching is a two-step process where the steps must be made compatible: Metadata prefetching, and Data block prefetching. Metadata prefetching from NameNode mitigates access latency on metadata and reduces NameNode overhead. Data block prefetching from DataNodes reduces I/O operation costs. In both steps, we identify some issues for efficient prefetch:

^*Corresponding author:Rana Ghazali

Prefetching time: At what point in time is it appropriate for the prefetch process to be triggered? If data blocks are cached before they are requested they will have little chance of being accessed. Due to limited cache space, there is a high probability that these data blocks will have been evicted from the cache at the time that they are needed. Therefore, we should determine a suitable prefetch time in terms of system parameters such as node processing capacity, task processing time, and on-demand data access time.
Prefetched data: Which data should be prefetched in each round? We should be aware of task-requested data in each step to prefetch on-demand data just before they are processed. If we prefetch the wrong data, on-demand data will not be accessed from the cache leading to high data access time.
Location of prefetched data: Where is a suitable location to store prefetched data? Data locality is an important factor that has significant impacts on decreasing job execution time via reducing data transmission time. In other words, we should take into account the shortest distance between the processing node and the location of the prefetched data. Therefore, the cache of the processing node is the ideal place. If the cache capacity of the processing node is full, caches of neighboring nodes can be considered.
Prefetched data volume: How much data should be prefetched in each step? By considering cache size, data block size, and node processing capacity, we would like to determine the suitable volume of data to prefetch. If these data volumes are too large the cache not only contains data items that have not been used recently but cache pollution may also arise as a result of the cache space used. Also, the frequency of prefetching would be increased if we consider low prefetched data volumes. This leads to poor performance as a result of increased overhead.

There are different lines of research that consider each of these issues in the prefetch mechanism individually, but we are not aware of any work that considers all of them together. In this paper, we present a smart prefetch strategy using KNN (K Nearest Neighbors) clustering [4] to determine which data (including volume) should be prefetched in each step. Moreover, we determine a suitable time to launch the prefetch mechanism by considering node processing capacity and task processing time. Finally, data locality considerations determine where prefetched data should be placed. Our contributions in this paper are:

We provide a brief overview of existing prefetch mechanisms and discuss their advantages and disadvantages.
We propose a novel prefetch mechanism that calculates the prefetch time and via the KNN cluster algorithm determines which data should be prefetched and where it should be placed by considering data locality.
We carry out experiments to investigate the impact of our proposed prefetch mechanism on Hadoop performance.

The rest of the paper is organized as follows: We discuss existing prefetch strategies for Hadoop and compare their advantages and disadvantages in Section 2. We then describe the proposed framework and present our KNN-based prefetch algorithm in Section 3. The design of the three steps of our prefetch algorithm is explained in Section 4 and its performance is evaluated via different experiments in Section 5. Finally, Section 6 contains conclusions and suggestions for future work.

Since BigData applications contain massive numbers of data blocks, there is no guarantee that all tasks can obtain their input data blocks from the cache. Due to a large amount of data, data blocks are likely evicted from the cache before they are accessed at demand time. In [5] Just Enough Cache (JeCache) was proposed as a solution for this problem using a just-in-time data block prefetching mechanism. This mechanism monitors data block access and calculates average data processing time to determine the minimal number of data blocks that should be kept in the cache. JeCache consists of two parts: 1) Prefetch information generation uses job history logs to determine which data blocks should be cached initially and a sequence of data blocks that need to be prefetched when a job is running. 2) Prefetch controller monitors data block access in each worker node, evicting data blocks from the cache when their processing is finished. This mechanism can reduce cache resource demand and guarantee efficient execution time however, it only considers read caching.

Vinutha et al. [6] introduced a solution to decrease data transmission time between a remote node and a processing node in a heterogeneous cluster. For this purpose, a prefetch thread is created to fetch requested input data in advance from a remote node to the buffer of the processing node, which is used as temporary storage. This results in a positive impact on job execution time by overlapping data transmission with data processing and increasing the data locality rate when launching a task. Even with this prefetching strategy, a job waits for the first data transmission.

In [7] a streaming technique was presented to solve this problem. In this method, data transfer and data processing are performed simultaneously. The resulting smaller size of the streaming data can reduce transmission waiting time.

Table 1

Comparison of prefetching strategies
Technique	Prefetch type	Location to store prefetched data	Advantages	Disadvantages
Jecache [5]	HDFS prefetch	DataNode cache	Uses cache space efficiently	Only considers read cache
Prefetch thread [6]	Node prefetch	Processing node buffer	Reduces transmission time	Waiting time for first data transmission
Streaming technique [7]	Node prefetch	Processing node buffer	Reduces waiting time	Does not load balance
Speculative prefetch [8]	Node prefetch	DataNode cache	Considers node processing capacity	Does not take into account node throughput
Two-level correlation base file prefetching [9]	HDFS prefetch	Considers four patterns (ND-pattern, CD-pattern, NC-pattern, and CC-pattern)	Dynamic replica selection	Does not consider some features

Kalia et al. [8] proposed speculative prefetching that takes into account the node processing capacity to load input data into the processing node. It groups intermediate data via the KNN clustering algorithm using a Euclidean distance measure to improve the data locality rate for Reduce tasks, leading to enhanced performance. However, it does not consider other features like workload capabilities and DataNode throughput.

In [9] a two-level correlation base file prefetching mechanism and dynamic replica selection were introduced to reduce data access latency and avoid overloaded DataNodes via load balancing. In this strategy, four placement patterns are considered to store fetched data.

Figure 1 depicts the workflow of the proposed prefetching mechanism that employs the KNN (K Nearest Neighbors) algorithm. In addition to the data block in demand, we also prefetch the K nearest data blocks as they are more likely to be needed soon.

Consider the following scenario, similar to that considered in [10]. When worker node N1 requests a block with ID 110, it first checks the cache metadata to determine whether the K closest neighboring data blocks to the requested block (IDs in the range 108–112 in our example, if K equals 4) are cached or not. If a cache miss occurs, a proactive cache request is initiated by asking the NameNode to locate replicas of those blocks. If replicas of those blocks are available on some Data Nodes, the NameNode requests the caching of these data blocks. In the illustrated example, a cache miss occurs for blocks 109 and 112, so a request to cache these blocks is sent to the NameNode. The NameNode selects the closest worker node to N1, in this case, N3 for block ID 109 and N4 for block ID 112, to reduce latency through data locality.

The present study aligns with the scenario introduced by Tahir et al. as detailed in their work [10], wherein the focus is on prefetching requested data blocks in conjunction with their adjacent data blocks to proactively prevent potential cache misses. However, a noteworthy distinction lies in the absence of K-nearest neighbors (KNN) clustering in their investigation, a technique that holds the potential to better determine the selection of data blocks and their respective volumes to be prefetched at each step. This study addresses this limitation by incorporating KNN clustering into the prefetching process, aiming to enhance the accuracy and efficiency of prefetching operations, thereby mitigating the likelihood of future cache misses.

In this section, we present the design of the proposed prefetching mechanism by addressing the key issues discussed in the introduction. Firstly, we determine when to launch the prefetching mechanism based on the progress rate of tasks. Secondly, based on the processing capacity of the worker node and the available cache space, we calculate the number of data blocks that can be prefetched in each round, K, using the KNN algorithm. Finally, we determine the location for prefetched data based on the data locality rate to reduce execution time by minimizing data transmission time

4.1 Prefetch time

In theory, the optimal timing for launching prefetching in Hadoop is determined by when the prefetched data can be fully utilized by Map or Reduce tasks, and when it can be made available in the cache before it is needed. This approach can help minimize the impact of data access latencies on Hadoop job performance. One method of determining the best timing is to use the progress rate of Map and Reduce tasks to predict when they will require data. In this case, a threshold value should be set for the progress rate of Map and Reduce tasks. Once the progress rate exceeds the threshold, the prefetching mechanism can be launched to begin fetching data blocks that are likely to be needed soon. By launching prefetching at this time, the data can be made available in the cache before it is needed, reducing data access latencies. To avoid interfering with other tasks or causing excessive network traffic, it is advisable to trigger prefetching when the system is relatively idle, such as during periods of low job activity or off-peak hours. Ultimately, the optimal timing for prefetching in Hadoop depends on various factors, including the job's characteristics, worker node processing capacity, and network resource availability. Monitoring system performance and adjusting the prefetching strategy as needed is critical to achieving favorable results.

To determine the best time to launch prefetching, we calculate the progress rate of tasks and determine a suitable threshold value. To calculate this threshold, we first introduce some notation: a job consists of T tasks (either Map tasks or Reduce tasks), with each task processing N < key, value > pairs, the number of processed pairs is M, and the task has completed L stages (for Reduce tasks, there are three stages: copy data phase, sort phase, and reduce phase). The progress score of the ith task, PS_i, is estimated based on the percentage of the task's < key, value > pairs that have been processed, as shown in Eq. 1 [11]. The average progress score of a job, PS_avg, is then calculated using Eq. 2. Furthermore, the progress rate of a task T can be computed based on how many < key, value > pairs are processed per second, given the task has run for T_r seconds, as shown in Eq. 3 [12]. By setting a suitable threshold value for the progress rate of Map and Reduce tasks, the prefetch mechanism can be triggered when the progress rate exceeds the threshold value, indicating that the task will soon require data blocks that can be prefetched.

PS_{i =} M/N for Map tasks (1)

PS_{i =}1/3(L + M/N) for Reduce tasks

PS_avg= (1/T) * \(\sum _{i=1}^{T}\text{P}\text{S}\)_i (2)

PR_i=PS_i/T_r (3)

To determine the threshold value for launching the prefetch mechanism, we need to consider the workload and system characteristics. The choice of threshold should balance the risk of triggering prefetching too early (leading to wasted network bandwidth and cache space) and the risk of triggering prefetching too late (resulting in longer processing times due to data access latency).

Our suggested formula for determining the threshold value is expressed as Eq. 4, where α and β are scaling factors that can be adjusted based on the characteristics of the workload and cluster. To determine α and β, one could consider workload features such as cache affinity and job type (I/O-bound and CPU-bound), while the cluster specifications could include available cache space and worker node processing capacity. To calculate the standard deviation of the progress rate of Map and Reduce tasks across the entire job, we use Eq. 5, where T is the total number of progress rate values. By taking into account these factors, we can determine an appropriate threshold value for launching the prefetch mechanism that minimizes the impact of data access latencies on the performance of Hadoop jobs.

Threshold = α * (PS_avg + β * PS_std) (4)

PS_std = sqrt [ \(\sum _{i=1}^{T}(\text{P}\text{S}\)_i- PS_avg)² / (T − 1) ] (5)

4.2 Prefetched data size

The size of prefetched data is a crucial factor affecting performance. Insufficient prefetched data can result in the system having to retrieve demand data from the disk or network, leading to increased data access time or an increased frequency of prefetch operations. On the other hand, excessive prefetched data can lead to an excessive load on a worker node and increased resource contention, negatively affecting performance. The size of data blocks that can be prefetched at each round depends on various factors, such as the processing capacity of the node, job running time, and the degree of parallelism. In a Hadoop heterogeneous environment, where the cluster consists of a set of worker nodes with varying processing capacities and cache spaces, the prefetched data size is calculated using Eq. 6. Here, W is the number of worker nodes, P_j denotes the processing capacity of the jth worker node, and C_j represents the available cache space on the jth worker node. The equation takes into account the parallelism degree, denoted by N_c, which refers to the number of tasks that can run concurrently, the estimated average processing time of a data block in the jth worker node, denoted by ET_j, and the data block size, denoted by BS.

K = \(\sum _{j=1}^{W}(\text{P}\)_j * ET_j * C_j )/(BS*N_c) (6)

4.3 Prefetched data location

The selection of the prefetched data location through the consideration of data locality measures can have a positive impact on decreasing job execution time. Assume that for a set of data nodes where all replicas of data blocks are located, D_d is the distance between data node d and the processing node. Additionally, C_d represents the processing capability of node d (a combination of CPU and memory capacity), while L_d represents the workload of data node d. It is important to choose an appropriate data node with a low load and distance to the processing node, to reduce data transmission time, and ensure enough processing capacity to balance the load in the cluster, thereby increasing resource utilization. A high-level conceptual formula that incorporates these parameters is given in Eq. 7, where f is a function that takes into account the specific algorithms and policies implemented in Hadoop to calculate a suitable location for prefetching data. It is obvious that data locality and node processing capacity have direct relationships, as Hadoop prioritizes nodes with local copies of the required data and that are closer to the nodes where the task is scheduled. Also, nodes with higher processing capacity are preferred for executing tasks to ensure that the computational load can be handled effectively. When a node is under heavy load or already processing a significant amount of data, it is logical to avoid placing additional prefetched data on that node and to distribute the tasks among less loaded nodes, ensuring a balanced workload distribution. In summary, the selected node for the prefetched data, S, is given by:

S = arg min{f ((D_d, C_d)/L_d)} (7)

A similar formula is used in the dynamic replica selection algorithm proposed in [9], where identical parameters are taken into account to determine which data blocks should be replicated, thereby mitigating the occurrence of hot DataNode phenomena. We do emphasize that the details of this choice (in particular the choice of the function f) may require some application-dependent customization.

4–4 Proposed prefetch algorithm

This section outlines our proposed algorithm for smart prefetching, which we present in detail. The input to our KNN prefetch algorithm includes a list of worker nodes in the cluster along with their processing capabilities, a list of tasks to be performed, and the size of the input data. First, we calculate the number of data blocks based on the Hadoop Distributed File System's (HDFS) data block size. Next, we estimate the time remaining for each task based on its progress, which depends on the task type, using Eq. 2. We then check whether the estimated time remaining for a task reaches the predefined threshold value or not. If it meets the threshold value, the prefetching process is initiated. The number of data blocks that can be prefetched, denoted by K, is determined using Eq. 6. We use K-nearest neighbor (KNN) clustering to group the data blocks into clusters, with each cluster containing the K-nearest neighbor data blocks that can be prefetched simultaneously during each prefetch phase. We determine the K data blocks that are located in the nearest neighborhood of the demand data blocks based on their Euclidean distance. Finally, we locate the prefetched data blocks based on the data locality factor, as described by Eq. 7, which reduces the data transmission time through the network and positively impacts the job execution time.

KNN prefetch algorithm

Input: List of heterogenous worker nodes, Input data size, List of tasks

1- TDB = IDS/DBS //Calculate total number of data blocks

2- If task T_i is Map task // Calculate the progress rate of the task

3- PS_{i =} M/N

4-else PS_{i =}1/3*(L + M/N)

5- End if

6-ProgressRate_i = PS _i/ExecutionTime_i

7-TimetoEnd_i = (1 − PS _i)/ProgressRate

8- PS_avg= (1/T) * \(\sum _{i=1}^{T}\text{P}\text{S}\)_i

9₋If TimetoEnd_i > = Threshold* PS_i then launch preftech

10- K = \(\sum _{j=1}^{w}(\text{P}\)_j * ET_j * C_j )/(BS*N_c) //Calculate the number of the data blocks that can be prefetch

11- For j = 1, j < IDS, j++

12- distance = distance + data[j]-data[j + 1]^2 // Calculate Euclidean distance for KNN

13- List= (SQRT(distance))

14- Sorted list = sort and group distance in ascending order

15- Choose the top K entries from the sorted list

16- Look up block metadata to find the location of these K data blocks

17- If they are not located in the DataNode then

18- DataNodes = arg min{f ((D_d, C_d)/L_d)}//Data locality to locate prefetched data

19- Send a request to the DataNodes to cache them

In this section, we explain the experimental environment including software and hardware configurations and some Hadoop configuration parameters settings. Our evaluation is divided into two sections: investigating the appropriate threshold value for choosing the best prefetch time and evaluating the impact of smart prefetch on Hadoop performance.

5.1 Experimental setup

For our experiments, we use a cluster consisting of a single NameNode and ten DataNodes located in two racks such that odd nodes are in rack1 and even nodes are in rack2.

Table 2

Node characteristics
Node type	Hardware configuration	Node processing capability
Master node	Intel Core i7-6700 processor, 2.4 GHz CPU, 64 GB memory, 10 data blocks cache capacity, and 1 TB hard disk.	8 Map slots, 5 Reduce slots
Odd slave nodes	Intel core i5-4590 processor, 3.30 GHz CPU, 32 GB memory, 8 data blocks cache capacity, and 500 GB hard disk.	4 Map slots, 2 Reduce slots
Even slave nodes	Intel core i5-4590 processor, 2.9 GHz CPU, 16 GB memory, 6 data blocks cache capacity, and 250 GB hard disk.	2 Map slots, 1 Reduce slot

Hardware configuration: The nodes are connected via a 10 Gigabit Ethernet switch. The experimental environment is a heterogeneous environment with different memory sizes and processing capabilities. Details are presented in Table 2.
Software configuration: We use the Ubuntu14.04 operating system and JDK 1.8, Hadoop version 2.7, and Intel HiBench [13] [14] version 7.1.
Hadoop configuration parameters: The block size of files in HDFS is 128 MB, and data replication is set to 3. Table 3 presents the Hadoop configuration parameters with their values. The remaining Hadoop configuration parameters are set to default values.
MapReduce applications: We use Intel HiBench as a Hadoop benchmark suite that contains the following applications: 1) WordCount is a CPU-intensive application that returns the number of occurrences of each word in a text file. 2) Sort is a typical I/O-bound application that sorts input data. 3) Grep is a mix of CPU-bound and I/O-bound operations that searches for a substring in a text file. In addition, the cache affinity feature [15] determines how to utilize the benefit of cached data in each application such that it can be classified into three categories based on this feature: low cache affinity (Sort), medium cache affinity (WordCount), and high cache affinity (Grep).
Input data: For carrying out experiments, we have used the default data sizes from the HiBench suite. Sort and WordCount have 60 GB and Grep has 1 TB for their input data sizes. The input data are generated by using the RandomTextWriter.

Table 3

Hadoop parameters
Hadoop property name	Hadoop property value
Dfs.replication	3
Dfs.blocksize	128M
Mapreduce.map.memory.mb	1024
Mapreduce.reduce.memory.mb	2048
MapReduce.jobhistory.address	Master:10020
MapReduce.jobhistory.webapp.address	Master:19888
MapReduce.reduce.speculative	False
MapReduce.map.speculative	False
Mapred.map.tasks.speculative.execution	False
Mapred.reduce.tasks.speculative.execution	False

5.2 Metrics

In our experiments, we consider two key performance metrics:

Job execution time: This plays a vital role in Hadoop performance improvement, and is related to data access time. The data access time decreases significantly if we could access data from the cache instead of the disk, reducing the job execution time.
Data locality rate: This is measured using the total number of tasks run locally in the node where the associated data resides.

5.3 Investigating the threshold value

As we mentioned in the previous section, selecting the appropriate value for the threshold parameter is a crucial issue that depends on various factors. For this purpose, we consider two workloads to investigate the impact of different threshold values on job execution time: WordCount as a CPU-oriented application with medium cache affinity, and Sort as an I/O-oriented application with low cache affinity.

In Fig. 2, we can observe that execution time increases when the threshold value is less than 0.6 or more than 0.8. The number of references to HDFS for fetching data blocks rises when the threshold value is too small. Prefetching large amounts of data blocks before requiring them leads to overload. In this case, there is a high probability that some data blocks are evicted from the cache before they are required, leading to a negative impact on job execution time. Also, if the threshold value is set too large (greater than 0.8) a greater number of data blocks are not accessible from the cache when required as a result of needing to wait until the prefetch mechanism is completed. The result is resource underutilization if prefetch is performed too late. Therefore, we should take into account the trade-off between these two issues. For our experiments, a suitable value is 0.7 independent of the application. It would be of interest to further explore if such insensitivity holds across other applications and/or operational environments.

5.4 Impact of smart prefetch on Hadoop performance

In this section, we evaluate the performance of smart prefetch based on two metrics: data locality rate and job execution time. For this purpose, we run a combination of the Micro benchmark in Intel HiBench that is composed of both I/O-bound jobs and CPU-bound jobs to compare our smart prefetch algorithm with Hadoop original and Hadoop with prefetch mechanism.

5.4.1 Performance evaluation based on data locality rate

Figure 3 presents the data locality rate for three applications and compares our smart prefetch algorithm with Hadoop original and Hadoop with prefetch. Smart prefetch has the best local tasks rate, demonstrating that it improves data locality by locating prefetched data in an intelligent manner. Also, by prefetching high-probability demand data just in time, more tasks have a chance to use cached data. Experimental results show that the locality rate of tasks has improved by 10.71% and 4.41% in WordCount, 11.90% and 7.14% in Sort, 12.19%, and 5.74% in Grep against Hadoop original and Hadoop with prefetch respectively. This suggests that our method is more suitable for high cache affinity applications and I/O-bound jobs, as they benefit more from cached data.

5.4.2 Performance evaluation based on job execution time

In this experiment, we use average job execution time as a performance measure to evaluate our proposed prefetching strategy. We use the Delay job scheduler that considers the data locality rate in its scheduling policy. We also execute each application 10 times to calculate the average execution time. In Fig. 4, we can observe that WordCount, Sort, and Grep finish their jobs 17%, 13.2%, and 12% faster than the default prefetch mechanism and their execution time improves by 30%, 25%, and 30% against Hadoop native. This novel prefetch mechanism improves the data locality rate with a resulting positive impact on execution time due to decreased data transmission time.

In this paper, we have presented a smart prefetch mechanism comprised of three phases. Firstly, we determine the optimal launch time for prefetching data blocks from the Hadoop Distributed File System (HDFS) by calculating the progress rate of tasks on worker nodes. This phase's success is limited by the threshold parameter, which must be set based on various environment and workload characteristics. While this paper provides a proof of concept of our prefetching algorithm, further work on how to choose this threshold is required. This could consist of additional guidelines, but ideally one would like to choose/learn this parameter in an automated fashion. Next, we utilize the K-Nearest Neighbor (KNN) algorithm to identify the number of data blocks with a high demand priority that can be fetched in each round. Finally, we evaluate data locality as a metric for placing the prefetched data. Experimental results indicate an average job execution time improvement of approximately 28.33% and 14% when compared to Hadoop and the default prefetch mechanism, respectively. This improvement is attributed to an 11.6% and 5.76% increase in data locality rate, respectively. Moving forward, we plan to assess the proposed prefetch mechanism's scalability by testing it on a large cluster.

Funding: Not applicable
Conflicts of interest/Competing interests: Not applicable

“Apache Hadoop” http://Hadoop. apache. org/
J. Dean and S. Ghemawat, MapReduce: Simplified Data Processing on Large Clusters, Communications of the ACM, vol. 51, no. 1,(2008)
K. Shvachko, H. Kuang, S. Radia, and R. Chansler, The Hadoop Distributed File System, IEEE 26th Symposium on Mass Storage Systems and Technologies (MSST), (2010)
H. Li, H. Jiang, D. Wang, B. Han, An improved KNN algorithm for text classification, Eighth International Conference on Instrumentation & Measurement, Computer, Communication and Control IMCCC, 2018, pp. 1081–1085. (2018)
Luo, Y., Shi, J. & Zhou, S. JeCache: Just-Enough Data Caching with Just-in-Time Prefetching for Big Data Applications. Proceedings - International Conference on Distributed Computing Systems 2405–2410 (2017) doi:10.1109/ICDCS.2017.268.
Vinutha, D. C. & Raju, G. T. Data Prefetching for Heterogeneous Hadoop Cluster. 2019 5th International Conference on Advanced Computing and Communication Systems, ICACCS 2019 554–558 (2019) doi:10.1109/ICACCS.2019.8728373.
Lee, J., Kim, K. T. & Youn-chen, T. MapReduce Performance Scaling Using Data Prefetching, 9, 26–31 (2022).
Kalia, K. et al. Improving MapReduce heterogeneous performance using KNN fair share scheduling. Robotics and Autonomous Systems 157, 104228 (2022).
Dong, B. et al. Correlation-based file prefetching approach for Hadoop. Proceedings - 2nd IEEE International Conference on Cloud Computing Technology and Science, CloudCom 2010 41–48 (2010) doi:10.1109/CloudCom.2010.60.
Singh, G., Chandra, P. & Tahir, R. A Dynamic Caching Mechanism for Hadoop using Memcached (2012)
Chen, Q., Zhang, D., Guo, M., Deng, Q. & Guo, S.: SAMR: a self-adaptive mapreduce scheduling algorithm in heterogeneous environment. In: Proceedings—10th IEEE International Conference on Computer and Information Technology, CIT-2010, 7^th IEEE International Conference on Embedded Software and Systems, ICESS-2010, ScalCom-2010. pp. 2736–2743 (2010). https://doi.org/10.1109/CIT.2010.458
Naik, N.S., Negi, A., Sastry, V.N.: Performance improvement of mapreduce framework in heterogeneous context using reinforcement learning. Procedia Comput. Sci. 50, 169–175 (2015)
Huang S, Huang J, Dai J, Xie T, Huang B (2014) The HiBench Benchmark Suite : Characterization of the MapReduce-Based Data Analysis. https://doi. org/ 10. 1109/ ICDEW. 2010. 54527 47
“Hibench” https:// github. com/ Intel- bigda ta/ HiBench
Kwak, J., Hwang, E., Yoo, T., Nam, B. & Choi, Y. In-memory Caching Orchestration for Hadoop. (2016) doi:10.1109/CCGrid.2016.73.

Competing interest reported. Job scheduling, distributed computing, big data, Machine learning

Download PDF

Version 1

posted

You are reading this latest preprint version

Smart Data Prefetching Using KNN to Improve Hadoop Performance

Status:

Version 1

Abstract

Figures

1 Introduction

2 Related work

3 Proposed workflow for the prefetching mechanism

4 Design of proposed prefetch mechanism

4.1 Prefetch time

4.2 Prefetched data size

4.3 Prefetched data location

4–4 Proposed prefetch algorithm

5 Smart prefetch evaluation

5.1 Experimental setup

5.2 Metrics

5.3 Investigating the threshold value

5.4 Impact of smart prefetch on Hadoop performance

5.4.1 Performance evaluation based on data locality rate

5.4.2 Performance evaluation based on job execution time

6 Conclusion and future work

Declarations

References

Additional Declarations

Status:

Version 1