IRPDP_HT2: A Scalable Data Pre-processing Method in Web Usage Mining using Hadoop-MapReduce

doi:10.21203/rs.3.rs-1323751/v1

Download PDF

Research Article

IRPDP_HT2: A Scalable Data Pre-processing Method in Web Usage Mining using Hadoop-MapReduce

https://doi.org/10.21203/rs.3.rs-1323751/v1

This work is licensed under a CC BY 4.0 License

Journal Publication

published 24 Mar, 2023

Read the published version in Soft Computing →

You are reading this latest preprint version

Data preparation is a vital step in the Web usage mining process since it provides structured data for the subsequent stages. Raw server logs should be turned into user sessions in order to generate structured data for pattern discovery phase. In recent decade, popular websites' server log production has risen to many terabytes to petabytes each day. As a result, server logs have Big Data challenges including storage and processing. Data cleaning, user identification, and session identification are data-intensive and deemed-computation-intensive operations in the data pre-processing sub phases. In recent years, Google's MapReduce parallel programming framework for data-intensive applications has grown in popularity as it provides the facility of scalable, flexible, fault-tolerant, distributed, and data-intensive processing over a cluster of nodes. In this paper, we provide IRPDP_HT2, a parallel data pre-processing algorithm using MapReduce that comprises data cleansing, user identification, and session identification approaches. The relevance and scalability of IRPDP_HT2 are also assessed using real web server logs, and it is discovered that the presented approach is both efficient and scalable.

Big Data

MapReduce

Hadoop

Data Preprocessing

Web Mining

Web Usage Mining

In this age of data expansion, studying users' navigation patterns is a critical responsibility for offering enhanced services to users. Web usage mining is a sort of Web mining that analyses user navigation patterns from server log data. Web usage mining patterns have been effectively implemented in a variety of applications, including personalization, prefetching and caching, website redesign, and business analytics [1, 2]. Web usage mining is segmented into three interdependent classes: data pre-processing, pattern discovery, and pattern analysis, and it generally follows the Knowledge Discovery in Databases process. Because of the distinctiveness of log data, data pre-processing is one of the most complicated and time-consuming processes in the Web usage mining process [3]. A massive amount of server log data is generated with over 1.7 billion websites and 3.7 billion internet users and their interactions with websites [4, 5]. Pattern discovery methodologies cannot be applied successfully to raw server log due to the unstructured format of logs. As a result, data preparation is a critical step in converting log data into a format that can be used by a certain data mining approach. Prior to pattern identification, using precise data preparation techniques ensures the efficiency and quality of subsequent stages of the Web usage mining. Data fusion, data cleansing, user identification, session identification, path completion, data transformation, and data formatting are sub-steps of data pre-processing [6]. Popular websites' log generation has expanded to many terabytes and petabytes each day in past few years. As a result, handling log files at such a scale is difficult and challenging. For instance, early 2010, the social media website Facebook handled with 130 terabytes of log data per day, that has now climbed to 300 petabytes per day in 2014 [7]. Conventional data processing methods are incapable of processing data at this scale, and log data suffers from Big Data challenges such as Volume, Velocity, and Variety, among others. Big Data technologies place a premium on not just storing and managing large amounts of data, but also extracting and interpreting it quickly [8]. The majority of data pre-processing sub tasks like data cleansing and user identification, are I/O-based rather than CPU-based. The techniques used in these phases spend more time on I/O than on data computation. Parallel processing is mandatory in the age of data bursts to handle massive amounts of data quickly [9]. Parallel programming architectures are divided into three categories: distributed memory, shared memory, and hybrid shared distributed memory. Processors with a shared memory architecture are able to share the same global memory space. OpenMP is a widely used multicore API for shared memory parallel design. OpenMP uses a more expensive scale-up hardware method than a scale-out hardware method. Data-intensive tasks, which are deemed computation intensive, work well with the scale-out technique. On the other hand, in a distributed memory system, each CPU has its own local physical address and interacts with one another via a local area network. MPI (Message Passing Interface) is a popular distributed computing programming model. Programmers are in charge of distributing data or tasks to nodes, managing node failures, and scheduling work in MPI. The MPI system model is based on the concept of transferring data to computation. Moving a huge amount of data via a network could be a barrier for data-intensive applications [10, 11]. The hybrid shared distributed memory architecture, which is the third parallel programming architecture, combines the capability of both shared memory and distributed shared memory architecture. For hybrid shared memory system, MPI with thread paradigm (OpenMP) is a widely used programming model [12]. Google's MapReduce is a fault-tolerant, scalable parallel programming model for handling huge datasets on a cluster of nodes [13]. Map Reduce is based on concepts such as a scale-out architecture strategy, moving computation to data, hiding system level information from developer, ability to handle faults, and seamless scalability [11]. Some open source implementations of MapReduce are Hadoop, Cascading, Sailfish, Disco, Skynet, FileMap, Themis etc. Among them, Hadoop is the most widespread and extensively used open-source MapReduce solution among them. In comparison to Hadoop, other MapReduce-based solutions might be thought of as low profile MapReduce runtime engines. To make development easier, these are either implemented on top of Hadoop or developed using script languages. Many Hadoop functionality, like as extensive APIs, diverse I/O format support, numerous compression algorithms, and a configurable scheduler, are also missing from these frameworks [14].

This study leverages the Hadoop-Map Reduce framework to design and develop scalable solutions for the data pre-processing stage of the Web usage mining process. A comprehensive Map Reduce-based data pre-processing method is built in this paper, which includes data cleansing, user identification, and session identification phases. For data cleansing two more parameters such as robots requests and internal dummy connections’ requests are considered. We assessed the effects of robots and internal dummy connection requests on session data by implementing fundamental session identification algorithm i.e. page stay time oriented heuristic. We investigated the correctness and scalability of this algorithm using an experiment on a cluster of nodes. It has been found that the implemented changes not only minimise the computation complexity for subsequent phases of the Web usage mining process, but also eliminate irrelevant session counts, enhance the effectiveness of session data in form of session count and duration.

The paper is structured as follows. Section 2 deliberates related work in server log analytics using Big Data technologies. The methodology and proposed algorithm is described in section 3. Experiments and result analysis are stated in section 4. Finally, section 5 states the conclusion.

Over the last few decades, log data has grown at an exponential rate, resulting in the problem of big data. The error analytics, statistical analysis, and pattern discovery phases of the Web usage mining process have received the greatest attention in log analytics research employing big data technology. Approaches to data pre-processing that use the Map Reduce framework have gotten less attention. Some of the server log analytics methodologies that make use of big data technologies are discussed in detail below.

Narkhede and Baraskar presented a HMR (Hadoop Map Reduce) log analyzer for banking Web server. Using the MapReduce framework, they calculated hits for the Web application, the city, and each page over the course of a year. This method reduces the amount of time it takes to retrieve statistical data [15]. Hingave and Ingle employed a NASA Web server log with 4, 45,454 records in another statistical analysis. They determined the number of queries per hour and the proportion of resources that were utilized. Furthermore, they examined their method to a relational database management system and noticed a reduction in response time [16]. Mavridis and Karatza had used the Hadoop and Spark frameworks to perform log analysis. They've created six modules for measuring daily requests, calculating individual requests, detecting prospective denial of service threats (dos attacks), recognizing dos offenders, measuring errors, and recognising the most frequent errors. HTTP status codes higher than 300 are used to record errors. They've tried a variety of things, including raising the number of data nodes, the quantity of the data, and the sort of application. In both systems, doubling the data nodes cuts the overall execution time in half, whereas doubling the input file doubles the runtime. Spark outperforms Hadoop in related to resource usage and reaction time [7]. To study users' navigation behaviour, Li et al. presented a frequent sequence mining approach that use the MapReduce programming model. Originally, they used a sequential technique to delete requests for image and script files, as well as unsuccessful queries, faulty queries, and irrelevant fields. In addition, three MapReduce tasks were utilized to generate regular sequence patterns. Sequence data is produced by using an IP address and a one-hour session time during first MapReduce task. A group of frequently visited pages is formed in the second MapReduce task. In addition, frequent sequence patterns are generated by filtering infrequent events. They've also tested the scalability of their methodology by expanding the quantity of the data and the number of data nodes [17]. Sisodia et al. devised a frequent sequence mining method, FP growth, employing the Spark architecture and showed that Spark is substantially faster than Hadoop [18]. Unlike navigation-based & time-based methods, Bayir et al. have suggested a Smart-SRA, which is an interconnected session identification method. It provides more comprehensive sessions that record more realistic user behaviour. Sessions are generated in two phases, depending on time ordering and topological criteria, accordingly. In the first stage of the process, which utilizes session time and page stay time, the server logs of visitors are distributed into short page sequences referred to as candidate sessions. The second stage produces the most sub-sessions from the candidate session using the topology rule and page dwell time. The proposed framework is also implemented using MapReduce framework. They have used one MapReduce job to implement session construction. They used IP addresses and domain names to identify users. Compared to earlier methodologies, their method yields more than 30% more accurate sessions [19, 20]. Data cleansing, on the other hand, is overlooked, and meaningless sessions are generated. Zhang and Zhang introduced data preparation options using MapReduce programming model. Data cleansing is applied to queries for the style sheet, image file, and script file, as well as queries with HTTP status codes below 200 and larger than 299 and HTTP access method other than GET. Moreover, IP address information is used to distinguish users, and a referrer-oriented heuristic is used to identify sessions [21]. Although their approaches are able to deal with large data but they lack potential robot detection approach and unable to resolve proxy server issue.

Savitha et al. have applied data cleansing, user identification, and session identification algorithms using just the Map Reduce framework. They choose fields like timestamp, number of bytes transferred, date, and version of browser for the further processing. They've adjusted the reduce task to zero, resulting in multiple cleaned log files based on the number of mappers that have been generated. The number of mappers would be larger for a bigger file, and the cleaned log would be placed in HDFS as a massive number of small files. Since there are fewer large files, Hadoop outperforms than when there are various small files [22]. Users are also recognised solely by their IP addresses, and sessions are determined using a time-oriented heuristic, i.e. session time [23]. Although their method can handle big datasets but it requires a method for detecting potential robots and confronts proxy server issues.

Huang et al. proposed a MapReduce-based technique for identifying users and sessions. ADSL and User agent fields have been used to differentiate clients. They used a customized referrer-based heuristic for session identification [24]. Their updated technique builds fewer big sessions than a referrer-oriented algorithm and can handle enormous datasets. However, they have not used any data cleansing techniques to minimize unwanted queries from the server log, which causes in uninteresting sessions during the session identification phase. Even though the abovementioned Map Reduce-based data pre-processing methods can manage large datasets, they cannot address proxy server and robot detection challenges. In a different approach, Srivastava et al. used the MapReduce platform to develop scalable data cleansing and user identification algorithms, which give efficient techniques for detecting ethical and unethical robots. Internal dummy connections' requests are taken into account in addition to typical data cleansing attributes. The strategy of matching User agent with the robots' aliases is used to detect ethical robots. These two changes have no effect on the data cleansing algorithm's time or space complexity [25]. To avoid proxy server problems, users are recognised using IP address and User agent fields in the user identification method. Additionally, when a user's browsing speed is high, their actions are rejected due to the possibility that they were produced by unethical robots [26].

As a better alternative, Srivastava et al. devised a MapReduce-based data pre-processing mechanism. This method includes data cleansing, user identification, and session identification as sub-phases of data preparation. To identify ethical and unethical robots, several effective algorithms are included into an existing MapReduce-based data pre-processing framework. According to several studies conducted on a cluster of nodes, their proposed approach is also flexible and effective for larger datasets. To see how efficient, the suggested strategy is, they looked at the impact of robot requests on sessions formed during the session identification phase [27].

One MapReduce job is employed in this strategy. A raw log file is initially imported into HDFS. It's divided into several blocks and delivered to data nodes based on block size and replication factor. Log entries are filtered using needed cleaning criteria and essential properties are picked in the Mapper phase of the method. Furthermore, cleaned items are classified according to the authentication of certain users and forwarded to the Reducer phase for processing. Following that, each user's activities are separated from those of robots. Users' activities are split into multiple sessions once the robots' activities are discarded. In addition, entries belonging to a certain session are saved in the session file. Following heuristics are used in these three phases: data cleansing, user identification, and session identification.

3.1 Data cleansing

Data cleansing is the procedure of removing from a log file any accesses or data that are not connected to the Web usage mining process [28]. There are lot of irrelevant information persist in server log file which are not required for analysing users ‘navigation behaviour. The size of the log file grows as a result of this raw data, which is an overhead for the future stages of the Web usage mining process. Six characteristics are examined for data cleansing: requests for irrelevant files, unsuccessful requests, unsuitable access methods request, internal dummy connections requests, robots requests, and reduction of unessential properties. When the HTTP access method field is absent, the request falls into the category of inappropriate access methods. Requests for robots are removed by following heuristics.

Heuristic 1

If Requested_url contains robots.txt and User agent log field does not have substring robots' aliases, preserve the requests for further phases to identify sessions formed by ethical robots that first request the robots.txt file while exploring the websites. Because the robots.txt file is sometimes not discovered on the Web server so requests for this file become unsuccessful requests and it is necessary to put robots.txt accesses in a cleansed log file explicitly. As a result, other cleaning criteria, such as the removal of unsuccessful requests, could be used to delete it.

Heuristic 2

Remove the request from the log file if User agent contains robots' aliases.

3.2 User Identification

Users are uniquely distinguished, and their activities are classified and listed in a user activity file during the user identification phase. Users' activities are classified in our method based on their IP address and User agent fields. Furthermore, if a user's activity includes access to the robots.txt file or if their browsing speed falls below a specific level, the user and their activities are removed as they belong to a robot.

3.3 Session Identification

Session identification is the practise of segmenting each user's behaviour into multiple sessions [29]. A user session, according to the W3C Web Usage Characterization Activity 1999, is a collection of activities taken by a user from the time he or she enters the website until he or she leaves it. To identify sessions, the following heuristic is used.

3.3.1 HT2: Time Oriented Heuristic

Total time spent on a particular Web page should not exceed by a threshold 𝜎. Let \(T1\) be a Timestamp of a request in the session \(S\) and next request with Timestamp \(T2\) will be added to session \(S\) if it satisfies the Eq. 1.

\(T2-T1\le {\sigma }\)

(1)

Otherwise a new session is created with timestamp (T2 +𝜎) [29].

The proposed algorithm i.e. IRPDP_HT2 based on above methodology is depicted in Alg. 1. A basic algorithm is also developed without consideration of internal dummy connections’ requests and robots requests. This algorithm is called PDP_HT2 as shown in Alg. 2. To evaluate the impact of internal dummy connections’ requests, raw log is cleaned with consideration of internal dummy connections’ requests at data cleansing phase and users are distinguished by IP address and User_agent information. After that, session identification approach HT2 is applied to identify sessions. We have called this algorithm as IPDP_HT2 as shown in Alg. 3. Further, to evaluate the impact of robots’ request, raw log is cleaned with consideration of robots’ requests at data cleansing phase and user identification phase by applying robots heuristics as discussed in section 3.1. After that, session identification approach is applied to extract identified sessions. This algorithm is called as RPDP_HT2 as shown in Alg. 4. Tab. 1 shows the description of symbols used in all these four algorithms.

Table 1

Description of Symbols used in Algorithms
Symbols	Descriptions
ip	IP address
t	Timestamp
a	HTTP_access_method
rq	Requested_url
s	HTTP_status_code
rf	Referrer_url
ua	User_agent
f_ext	File extension of Requested_url
comb_value	Combined requests of a user with a delimiter
i_t	Timestamp of the first request of a user
c_t	Timestamp of the current request of user
l_t	Timestamp of the last request of a user
s_t	Timestamp of the first request of a session
e_t	Timestamp of the last request of a session
prev_t	Timestamp of the previous request of a session
p_c	Number of requested pages
c_url	Combined requested urls of user with a delimiter
c_val	Current value for particular iteration
Bs	Browsing speed per page for particular user
U_ID	Unique user id for particular user
S_ID	Session Id

Algorithm 1: A MapReduce-based data pre-processing algorithm with consideration of robots’ requests and internal dummy connections’ requests (IRPDP_HT2)

Algorithm 2: A MapReduce-based data pre-processing algorithm without consideration of internal dummy connections’ requests and robots requests (PDP_HT2)

Mapper (key, value)

1: for each value

2: extract s, a, rq, ip, ua, by parsing value

3: if (s ∈ [200-299]) & (a ∈ [GET, POST]) &

(f_ext ∉ [multimedia file extensions]) & (ip ∉ [::1, 127.1.1.1, 127.0.0.1]) & (ua does not contain keywords [robot aliases])

4: extract t, rf by parsing value

5: key’: composite_key (ip, ua, t)

6: value’: combination of t, rq, & rf

7. EmitIntermediate (key’, value’)

8: end if

9: if (rq contains robots.txt) & (ua does not

contain keywords [robot aliases])

10: extract t, rf by parsing value

11: key’: composite_key (ip, ua, t)

12: value’: combination of t, rq, & rf

13: EmitIntermediate (key’, value’)

14: end if

15: end for

Reducer_HT2 (key, values)

1: for each value in values

2: Calculate p_c

3: end for

4: if (p_c equals to 1)

5: Generate U_ID, S_ID

6: value’: U_ID, S_ID, c_val

7: emit (key’, value’)

8: else

// When number of pages are more than 1 in user activity record

9: s_c = 1

10: for each value in values

11: extract c_t from value

12: c_val: value

13: if (s_c= 1)

// Create first session of user

14: Generate U_ID, S_ID

15: value’: U_ID, S_ID, c_val

16: emit (key’, value’)

17: prev_t = c_t

18: else if (c_t-prev_t <= 𝝈)

//Add request to existing session

19: emit (key’, c_val)

20: prev_t = c_t

21: else

//Create new Session for same user

22: s_c++

23: Generate S_ID

24: value’: U_ID, S_ID, c_val

25: emit (key’, value’)

26: p_t = c_t

27: end if

28: end for

29: end if

Mapper (key, value)

1. for each value

2. extract s, a, rq, ip, ua, by parsing value

3. if (s ∈ [200-299]) & (a∈ [GET, POST])

& (f_ext ∉ [multimedia file extensions])

4. extract t, rf by parsing value

5. key’ :composite_key (ip, ua, t)

6. value’: combination of t, rq, & rf

7. EmitIntermediate (key’, value’)

8. end if

9. end for

Reducer_HT2 (key, values)

1. for each value in values

2. Calculate p_c

3. end for

4. if (p_c equals to 1)

5. Generate U_ID, S_ID

6. value’: U_ID, S_ID,

c_val

7. emit (key’, value’)

8. else

// When number of pages are more than 1 in user activity record

9. s_c = 1

10. for each value in values

11. extract c_t from value

12. c_val value

13. if (s_c = 1)

// Create first session of user

14. Generate U_ID, S_ID

15. value’: U_ID, S_ID, c_val

16. emit (key’, value’)

17. prev_t = c_t

18. else if (c_t-prev_t <= 𝝈 )

//Add request to existing session

19. emit (key’, c_val)

20. prev_t = c_t

21. else

//Create new Session for same user

22. s_c++

23. Generate S_ID

24. value’: U_ID, S_ID, c_val

25. emit (key’, value’)

26. p_t = c_t

27. end if

28. end for

29. end if

Algorithm 3: A MapReduce-based data pre-processing Algorithm with consideration of internal dummy connections’ requests and without robots’ requests (IPDP_HT2)

Algorithm 4: A MapReduce-based data pre-processing Algorithm with consideration of robots’ requests and without internal dummy connections’ requests (RPDP_HT2).

Mapper (key, value)

1. for each value

2. extract s, m, r, ip, ua, by parsing

value

3. if (s ∈ [200-299]) & (a ∈ [GET,

POST]) & (f_ext ∉ [multimedia file extensions]) & (ip ∉ [::1, 127.1.1.1,

127.0.0.1])

4. extract t, rf by parsing value

5. key’: composite_key (ip, ua, t)

6. value’: combination of t, rq, &

7. EmitIntermediate (key’, value’)

8. end if

9. end for

Reducer_HT2 (key, values)

1: for each value in values

2: Calculate p_c

3: end for

4: if (p_c equals to 1)

5: Generate U_ID, S_ID

6: value’: U_ID, S_ID, c_val

7: emit (key’, value’)

8: else

// When number of pages are more than 1 in user activity record

9: s_t = 1

10: for each value in values

11: extract c_t from value

12: c_val: value

13: if (s_c = 1)

// Create first session of user

14: Generate U_ID, S_ID

15: value’: U_ID, S_ID, c_val

16: emit (key’, value’)

17: p_t = c_t

18: else if (c_t-prev_t <= 𝝈 )

//Add request to existing session

19: emit (key’, c_val)

20: prev_t = c_t

21: else

//Create new Session for same user

22: s_c++

23: Generate S_ID

24: value’: U_ID, S_ID, c_val

25: emit (key’, value’)

26: prev_t = c_t

27: end if

28: end for

29: end if

Mapper (key, value)

1: for each value

2: extract s, m, r, ip, ua, by parsing value

3: if(s ∈ [200-299]) & (m ∈ [GET, POST])

& (f_ext ∉ [multimedia file extensions]) & (ua

does not contain keywords [robot aliases])

4: extract t, rf by parsing value

5: key’: composite_key(ip, ua, t)

6: value’: combination of t, rq, & rf

7: EmitIntermediate (key’, value’)

8: end if

9: if (rq contains robots.txt) & (ua does not

contain keywords [robot aliases])

10: extract t, rf by parsing value

11: key’ :composite_key(ip, ua, t)

12: value’: combination of t, rq, & rf

13: EmitIntermediate (key’, value’)

14: end if

15: end for

Reducer_HT2 (key, values)

1. Extract i_t

2. for each value in values

3. Calculate comb_url, l_t & p_c

4. end for

5. if (comb_url contains robots.txt)

6. discard user // robot

7. else if (p_c equals to 1)

8. Generate U_ID, S_ID

9. value’: U_ID, S_ID,c_val

10. emit (key’, value’)

11. else

// When number of pages are more than 1 in user activity record

12. bs = (l_t - i_t + 1 )/p_c

13. end if

14. if (bs < θ)

15. discard user

16. else

17. s_c = 1

18. for each value in values

19. extract c_t from value

20. c_val: value

21. if (s_c== 1)

// Create first session of user

22. Generate U_ID, S_ID

23. value’: U_ID, S_ID, c_val

24. emit (key’, value’)

25. prev_t = c_t

26. else if (c_t - prev_t <= 𝝈 )

//Add request to existing session

27. value’: c_val

28. emit (key’, value’)

29. prev_t = c_t

30. else

//Create new Session for same user

31. s_c++

32. Generate S_ID

33. value’: U_ID, S_ID, c_val

34. emit (key’, value’)

35. prev_t = c_t

36. end if

37. end for

38. end if

All of these algorithms, IRPDP HT2, PDP HT2, IPDP HT2, and RPDP HT2, are developed in JAVA and use the MapReduce API of the Hadoop 2.6.0 framework. These are run on a four-node cluster, three-node cluster, and two-node cluster, respectively. In this cluster, one node serves as the master, while the others serve as data/slave nodes, each with four cores. Ubuntu 14.04 operating system is installed on all nodes. Real-world Apache Web server access logs from Banaras Hindu University's Web server are collected for experimentation. These log files have been combined into a single file. Furthermore, using a data extraction algorithm, datasets D1-D4 of sizes 0.72GB, 1.44GB, 2.88GB, and 5.76GB are recovered from a combined log file. The number of entries in these datasets are 3361728, 6725269, 13282787, or 26057551 respectively without duplicates. The results of the suggested MapReduce - based approach are explained in depth further below as relevancy analysis and scalability analysis.

4.1 Relevancy Analysis

In this section, we have analysed the impact of applied modifications i.e. internal dummy connections’ requests and robots’ requests at data cleansing and user identification phase on session data in terms of session count and session length. We have observed the impact of internal dummy connections’ requests and robot’s requests based on session count and session length. The page stay threshold 𝝈 is set to 10 minutes and browsing speed 2 seconds per page. Fig. 1 shows the comparison of session count with or without consideration of internal dummy connections requests and robots requests. From the graph, it is clear that robots’ requests and internal dummy connections’ requests not only reduces calculation overhead i.e. the number of requests but also removes the uninteresting session count for all three session identification heuristics. This improves the quality of session data for pattern discovery phase. The impact of robots’ requests is more than internal dummy connections’ requests because there are multiple types of robots identified hence results in various sessions. On the other hand, internal dummy connections’ requests are generated by a single server that results in fewer sessions Tab. 2 shows the comparison of session length with or without internal dummy connections’ requests and robots requests. Here, session length is classified into four categories according to their number of accessed pages in that session i.e. short sessions that have number of pages 1, medium sessions that have the number of pages from 2 to 10, long sessions that have the number of pages from 11-100, and extreme having the number of pages greater than 100. It can be observed experimentally that consideration of robots’ requests impact is more on extremely long sessions for all three heuristics. This improves quality of session data for pattern discovery phase.

Table 2

Impact of robots’ requests and internal dummy connections’ requests on session length by HT2 Session identification heuristic
Session identification heuristic	Session length type	Session count for PDP	Session count for IDP (Percentage of reduction w.r.t to PDP)	Session count for RPDP (Percentage of reduction w.r.t to PDP)	Session count for IRPDP (Percentage of reduction w.r.t to PDP)
D1	short	49247	49239 (0.016%)	46306 (5.972%)	46298 (5.988%)
	medium	84531	84521 (0.012%)	80445 (4.834%)	80435 (4.846%)
	long	10528	10525 (0.028%)	10108 (3.989%)	10105 (4.018%)
	extreme	219	215 (1.826%)	166 (24.201%)	162 (26.027%)
D2	short	80822	80782 (0.049%)	75687 (6.353%)	75926 (6.058%)
	medium	145743	145684 (0.04%)	138088 (5.252%)	137099 (5.931%)
	long	48086	48072 (0.029%)	46065 (4.203%)	45712 (4.937%)
	extreme	393	378 (3.817%)	272 (30.789%)	261 (33.588%)
D3	short	156592	156488 (0.066%)	145854 (6.857%)	144716 (7.584%)
	medium	278547	278329 (0.078%)	263328 (5.464%)	261834 (6%)
	long	74714	74664 (0.067%)	70998 (4.974%)	70558 (5.563%)
	extreme	652	623 (4.448%)	418 (35.89%)	390 (40.184%)
D4	short	296894	296648 (0.083%)	276284 (6.942%)	274642 (7.495%)
	medium	559823	559254 (0.102%)	526534 (5.946%)	521471 (6.851%)
	long	157219	157068 (0.096%)	149371 (4.992%)	148256 (5.701%)
	extreme	1184	1122 (5.236%)	697 (41.132%)	641 (45.861%)

4.2 Scalability Analysis

Two measures are used to assess the scalability of the IRPDP HT2 algorithm: By increasing Data nodes and by increasing the data sizes.

4.2.1 Impression of Increasing Number of Data Nodes

The algorithm IRPDP_HT2 is evaluated on 1, 2, and 3 data nodes respectively for all datasets D1-D4. The number of mappers are set to 3, 6, 12, and 24 for D1 to D4 datasets correspondingly. Fig. 2 shows the impact of increasing data nodes on IRPDP_HT1 for all four datasets. From graph, we can state that the larger datasets like D3 and D4 give better speedup performance as compared to D2 and D1 datasets. In the MapReduce system, job distribution and management takes longer than processing time for smaller datasets. However, with larger datasets, processing time is proportional to the size of the dataset, which can be lowered using parallelism. For large datasets, this processing time might be lowered by increasing the number of data nodes because the number of splits would be larger and the algorithm would be able to employ all cores of the data nodes. On expanding data nodes, the IRPDP HT2 algorithm scales well for larger datasets.

4.2.2 Impression of Increasing Data Sizes

The number of data nodes is fixed at three, while data sizes are doubled to test size-up performance. Fig. 3 shows that even after doubling the input data, the algorithm's running time does not double. Due to the MapReduce framework's efficient calculation, running time for larger data sets is not long.

We propose a parallel MapReduce-based data pre-processing algorithm (IRPDP_HT2) in this paper, which takes into account six parameters for data cleansing: unsuccessful requests, irrelevant files' requests, internal dummy connections' requests, robots' requests, unessential access methods' requests and reduction of unessential attributes. IP addresses and user agents are used to identify users. The page stay oriented heuristic is used to identify sessions for session identification. We tested an algorithm for performing relevancy and scalability analyses. On the previous two phases, data cleansing and user identification, we saw the impact of robot requests and internal dummy connections' requests in relevancy analysis. Internal dummy connections' and robots' requests are taken into account, which not only reduces irrelevant requests but also removes uninteresting session counts. The removal of robot queries has a greater impact on exceptionally long sessions, which improves the quality of session data for pattern identification. We calculated the Speedup and Sizeup metrics for the provided technique to determine scalability, and it delivers better Speedup and great Sizeup for larger datasets.

Acknowledgement: We are thankful to Dr. R. S. Yadav, Programmer, Computer Center, Banaras Hindu University for providing the server log Banaras Hindu University website.

Funding: The authors declare that no funds, grants, or other support were received during the preparation of this manuscript.

Competing Interests: The authors have no relevant financial or non-financial interests to disclose.

Conflicts of Interest: The authors declare that they have no conflicts of interest to report regarding the present study.

Data Availability: The datasets generated during and/or analyzed during the current study are not publicly available due to it is server logs of Banaras Hindu University website that contains some privacy information of Web users.

R. Cooley, B. Mobasher and J. Srivastava, “Web mining: Information and pattern discovery on the world wide web”, in Proc. IEEE international conference on tools with artificial intelligence, pp. 558–567, 1997.
F. M. Facca and P. L. Lanzi, “Mining interesting knowledge from weblogs: a survey”, Data & Knowledge Engineering, vol. 53, no. 3, pp. 225–241, 2005.
B. Liu, “Web data mining: exploring hyperlinks, contents, and usage data”, in Data-Centric System and Applications, 2^nd ed., vol. 1. Dordrecht, London, New York: Springer Press, 2011.
WebServerSurvey,https://news.netcraft.com/archives/2020/08/26/august-2020-web-server-survey.html, August 2020, accessed on September 2020.
Internet Statistics 2020, https://hostingfacts.com/internet-facts-stats, 2020, accessed on September 2020.
R. Cooley, B. Mobasher NS J. Srivastava, “Data preparation for mining world wide web browsing patterns”, Knowledge and information systems, vol. 1, no. 1, PP. 5–32, 1999.
I. Mavridis and H. Karatza, “Performance evaluation of cloud-based log file analysis with Apache Hadoop and Apache Spark”, Journal of Systems and Software, vol. 125, pp. 133–151, 2017.
B. Kotiyal, A. Kumar, B. Pant and R. H. Goudar, “Big data: Mining of log file through Hadoop”, in Proc. International Conference on Human Computer Interactions (IEEE), Chennai, India, pp. 1–7, 2003.
K.-H. Lee, Y.-J. Lee, H. Choi, Y. D. Chung and B. Moon, “Parallel data processing with MapReduce: a survey”, in ACM SIGMOD Record, vol. 40, no. 4, pp. 11–20, 2012.
S. J. Kang, S. Y. Lee and K. M. Lee, “Performance comparison of OpenMP, MPI, and MapReduce in practical problems”, Advances in Multimedia, vol. 2015, no. 7, pp. 7-16, 2015.
J. Lin and C. Dyer, “Data-intensive text processing with MapReduce”, in Synthesis Lectures on Human Language Technologies, 1^st ed., vol. 3, no. 1, Morgan & Claypool Press, 2010.
B. Barney, “Introduction to parallel computing”, Lawrence Livermore National Laboratory, vol. 6, no. 13, pp. 10, 2010.
J. Dean and S. Ghemawat, “MapReduce: simplified data processing on large clusters”, Communications of the ACM, vol. 51, no. 1, pp. 107–113, 2008.
F. Li, B. C. Ooi, M. T. Özsu and S. Wu, “Distributed data management using MapReduce”, ACM Computing Surveys (CSUR), vol. 46, no. 3, pp. 1–42, 2014.
S. Narkhede and T. Baraskar, “HMR log analyzer: Analyze web application logs over Hadoop MapReduce”, International Journal of UbiComp, vol. 4, no. 3, pp. 41-51, 2013.
H. Hingave and R. Ingle, “An approach for MapReduce based log analysis using Hadoop”, in Proc. International Conference on Electronics and Communication Systems (ICECS), pp. 1264–1268, 2015.
M. Li, X. Yu and K. H. Ryu, “MapReduce-based web mining for prediction of web-user navigation”, Journal of Information Science, vol. 40, no. 5, pp. 557–567, 2014.
D. S. Sisodia, V. Khandal and R. Singhal, “Fast prediction of web user browsing behaviours using most interesting patterns”, Journal of Information Science, vol. 44, no. 1, pp. 74–90, 2018.
M. A. Bayir, I. H. Toroslu, A. Cosar and G. Fidan, “Smart miner: a new framework for mining large scale web usage data”, in Proc. of the 18th international conference on World wide web, pp. 161–170, 2009.
M. A. Bayir, I. H. Toroslu, M. Demirbas, and A. Cosar, “Discovering better navigation sequences for the session construction problem”, Data & Knowledge Engineering, vol. 73, pp. 58–72, 2012.
G. Zhang and M. Zhang, “The algorithm of data preprocessing in Web Log mining based on cloud computing”, in Proc. International Conference on Information Technology and Management Science (ICITMS), pp. 467–474, 2013.
T. White, “Hadoop: The definitive guide” in MapReduce for the Cloud, 3^rd Edition, Yahoo Press, O’Reilly Media, Inc., 2012.
K. Savitha and M. S. Vijaya, “Mining of web server logs in a distributed cluster using big data technologies”, International Journal of Advanced Computer Science and Applications (IJACSA), vol 5, no 1, 2014
P. Huang, D. Chen, and J. Le, “An improved referrer-based session identification algorithm using MapReduce”, in Proc. International Conference on Natural Computation (ICNC), pp. 1072–1076, 2013.
M. Srivastava, R. Garg and P. K. Mishra, “Mapreduce-Based Parallel Data Cleaning Algorithm in Web Usage Mining”, International Journal of Computer Science & Applications, vol. 14, no. 2, 2017.
M. Srivastava, R. Garg and P. K. Mishra, “A mapreduce-based user identification algorithm in Web usage mining”, International Journal of Information Technology and Web Engineering, vol. 13, no. 2, pp. 11–23, 2018.
M. Srivastava, A. K. Srivastava, R. Garg and P. K. Mishra, “Performance Evaluation of the MapReduce-based Parallel Data Preprocessing Algorithm in Web Usage Mining with Robot Detection Approaches”, IETE Technical Review, pp. 1–15, 2021
Z. Pabarskaite, “Implementing advanced cleaning and end-user interpretability technologies in web log mining”, in Proc. International Conference on Information Technology Interfaces (IEEE), pp. 109–113, 2002.
M. Spiliopoulou, B. Mobasher, B. Berendt and M. Nakagawa, “A framework for the evaluation of session reconstruction heuristics in web-usage analysis”, Informs journal on computing, vol. 15, no. 2, pp 171–190, 2003.

Download PDF

Journal Publication

published 24 Mar, 2023

Read the published version in Soft Computing →

Reviewers agreed at journal
27 Aug, 2022
Reviewers invited by journal
21 Aug, 2022
Editor assigned by journal
07 Feb, 2022
First submitted to journal
03 Feb, 2022

You are reading this latest preprint version

IRPDP_HT2: A Scalable Data Pre-processing Method in Web Usage Mining using Hadoop-MapReduce

Status:

Journal Publication

Version 1

Abstract

Figures

1 Introduction

2 Literature Review

3 Methodology And Proposed Algorithm

3.1 Data cleansing

3.2 User Identification

3.3 Session Identification

3.3.1 HT2: Time Oriented Heuristic

4 Experimental Analysis

4.1 Relevancy Analysis

4.2 Scalability Analysis

4.2.1 Impression of Increasing Number of Data Nodes

4.2.2 Impression of Increasing Data Sizes

5 Conclusion

Declarations

Data Availability: The datasets generated during and/or analyzed during the current study are not publicly available due to it is server logs of Banaras Hindu University website that contains some privacy information of Web users.

References

Status:

Journal Publication

Version 1