Hadoop Distributed File System for Big data analysis

doi:10.21203/rs.3.rs-2125174/v1

Download PDF

Method Article

Hadoop Distributed File System for Big data analysis

https://doi.org/10.21203/rs.3.rs-2125174/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

Hadoop is framework that is processing data with large volume that cannot be processed by conventional systems. Hadoop has management file system called Hadoop Distributed File System (HDFS) that has NameNode and DataNode where the data is divided into blocks based on the total size of dataset. In addition, Hadoop has MapReduce where the dataset is processed in Mapping phase and then reducing phase. Using Hadoop for big data analysis has been revealed important information that can be used for analytical purpose and enabling new products. Big data could be found in many different resources such as social networks, web server logs, broadcast audio streams and banking transactions. In this paper, we illustrated the main steps to setup Hadoop and MapReduce. The illustrated version in this work is the latest released of Hadoop 3.1.1 for big data analysis. A simplified pseudo code is provided to show the functionality of Map class and reduce class. The developed steps are applied with a given example that could be generalized with bigger data.

Hadoop

MapReduce

HDFS

DataNode

NameNode

Big Data Analysis

Increasing the dependency on technology is generating immerse volume of data. However, process and analysis of large amount of data is difficult via conventional methods. Therefore, Apache Hadoop is designed to store and process data that has various size up to terabytes, petabytes and zettabytes [2]. Furthermore, applying big data analysis has been highly suggested to process the dataset by using Hadoop and MapReduce in order to enhance the computational and mining process [1].

A. Hadoop and Big Data

Many studies have been defining and explaining the big data as a large volume of data. In addition, velocity, variety and increasing the volume of data are other important features of big data [2]. In contrast, data mining is more about strategic decision making on structured data that can be as small volume of data or large volume of data [6].

There are different types of big data which are structured data, unstructured data and semi structured data [10]. Also, images, video, audio and natural language are considered as other types of big data [9].

Firstly, big data that has recognize pattern and easy to read and understand is recognized as structured data such as excel table or table in database [10]. Secondly, big data that has unrecognized pattern such as email file which is recognize as unstructured data [10]. Thirdly, semi-structured data that is recognized as data similar to XML and JSON [10].

The described types of big data are considering as valuable sources for big data analysis by using Apache Hadoop. In addition, real world example of applying Apache Hadoop as solution for big data analysis are highlighted as following [11]:

In Healthcare domain: Apache Hadoop provides analysis for archiving claims data and process for logging terabytes of data that are made from transactional system.
In Telephone Company domain: Apache Hadoop with HBase provides storage for billions of rows of call records details, where data with TBs are added on monthly basis.

The applications of Hadoop in real world are various. In fact, the variety, velocity and volume of data are generated rapidly on daily basis. Therefore, it is important to highlight in this paper the theory and concepts of Apache Hadoop and MapReduce as big data analysis tools as well as the main steps to setup Apache Hadoop ver.3.1.1 on Mac operating system.

B. Hadoop Architecture

Apache Hadoop is open-source software that is distributed processing of large data among multiple node or single node [3]. Hadoop installation is dependent on the aim of the project and the volume of data in dataset. User of Hadoop has the capability to process the large volume of dataset in multiple nodes, which is required Hadoop installation as pseudo-distributed model.

Alternatively, local mode or standalone mode allow the user of Hadoop to have single node on local system to process the dataset. The core components of Hadoop are recognized as Hadoop Distributed File System (HDFS) and MapReduce [1]. The HDFS is implemented in Java and mainly it is dividing the file of dataset into blocks based on the dataset size [10]. The HDFS process the dataset by using NameNode system and DataNode system [10]. The NameNode has main role in containing the dataset file and other related information such as dataset file name, permission and the location of each block of dataset [12]. Consequently, the NameNode is connected to DataNode where a replication of data blocks are stored in memory and processed for the read and write operations (Figure 1)[12].

C. Hadoop prerequisites

There are different methods to run Hadoop in different platforms. But most of the research articles and resources are depending on the Unix operating system [7]. Based on that, Linux based operating system is used to illustrate the steps for setup Apache Hadoop. In addition, there are several Hadoop versions available to download each version has different setup and configuration [8]. The latest version of Hadoop 3.1.1 is used to illustrate the setup and the remaining steps for big data analysis.

Installing Hadoop is requiring step by step process where the dependencies between steps are critical [3]. There are different methods to setup Hadoop based on the projects of big data analysis. One of the methods is Single Node Cluster where the Hadoop is processing the data on local system. Moreover, all methods for setup Hadoop are requiring installing Java version 8 (Figure 2).

Furthermore, SSH localhost should be open and the sharing option is enabled (Figure 3).

It is important to ensure that Java is installed and SSH is not blocked on port 22 (Figure 2) (Figure 3) [3]. Downloading the required Hadoop version and extracting the files can be complicated but using package management software such as Homebrew is simplifying the installation of software and ensure the latest update is installed [5]. Homebrew is simplifying the installation of Hadoop and it is ensuring the latest update for any other packages on Hadoop. Simply, type the following line in bash terminal in order to install Homebrew (Figure 4).

D. Hadoop Setup

In order to start the process of installing Hadoop type the following command in bash terminal (Figure 5).

Installing Hadoop is requiring more than 700 MB size from hard disk space and it may be required to install additional package such as Hive and Pig in order to simplify the analysis on big data analysis (Figure 6). The advantage of having homebrew is easier of installing other packages / software by typing the following syntax

$ brew install (type the name of software)

Once setup of Hadoop is completed, it shows the following

In case of removing Hadoop type the following

$ brew uninstall Hadoop (or type the name of software)

E. Hadoop Configuration

Hadoop has several files need to be configured before run Hadoop as single node mode [3]. There are two main directories need to be configured which are sbin and bin.

The first directory sbin has five files that need to add the following command (Figure 7)(Figure 8)(Figure 9)(Figure 10).

$ cd /usr/local/cellar/hadoop/3.1.1/libexec/etc/hadoop

The second directory bin has several files that are support the control of HDFS as the following path

$ cd /usr/local/cellar/hadoop/3.1.1/ libexec/bin

Simply, it is required to process a format for NameNode only for the first time after Hadoop installation as following command shows

$ hdfs namenode -forma

F. Running Hadoop

Access the directory sbin in order to start dfs

$ cd /usr/local/cellar/hadoop/3.1.1/ libexec /sbin

$ ./start-dfs.sh

Starting the dfs is lunching name node, data node and secondary name node, which are main components of HDFS.

$ ./start-yarn.sh

Starting the yarn is lunching the other parts of Hadoop, which are node managers and resource manager.

In order to check the functionality and ensure all components of Hadoop are lunched, type the following command (Figure 11)

Open the web browser http://localhost:9870, where it shows the NameNode page.

Open the web browser http://localhost:8088/cluster to access Hadoop main page.

MapReduce is software framework that is considered as part of Hadoop framework. It has the capability to process vast volume of data in terabyte that are available in several thousands of nodes [4]. MapReduce and HDFS are main components of Hadoop. HDFS is basically stored the data and using write and read operation model to distribute the data in local node (Single Node model) or distribute the data across multiple node (Multiple Nodes Model). Mainly, MapReduce is applying the map function on the required objects then applying the reducers function on the unwanted objects (Fig. 12)[4]. The functionality of MapReduce is based on programs written in various languages such as Java, C or Python [12]. Generally, most of the MapReduce programs are sharing the same process which are input split, mapping, shuffling, reducer then final output [4]. The mapping and reducing functionality are occurred on local model as well as on Pseudo-Distributed model with association from Yet Another Resource Node (YARN) [4].

Part of Hadoop is MapReduce function where the content of dataset is rearranging. The rearranging of objects in dataset is based on written program that has mainly Mapping function and then Reducer function [4]. During the process of MapReduce, the key and value rule is applied on the target objects in order to implement the Map process and Reduce process (Fig. 13)[4].

Firstly, common prerequisites of running MapReduce framework are ensuring the Hadoop is installed, configured and running [4]. MapReduce is processing the dataset from certain input path that user has to specified before the running MapReduce function (Fig. 13)(Fig. 16). The objects are processed based on written programs that mainly contain class for Mapping objects and another class for Reducing objects [4](Fig. 14).

A. Testing MapReduce

Due to dependence between Hadoop and MapReduce, user has to ensure Hadoop is running then change directory to the in bin directory and apply the following command to ensure MapReduce is working (Fig. 15).

B. MapReduce-use case

MapReduce and HDFS are main components of Hadoop. HDFS is basically stored the data and using write and read operation model to distribute the data in local node (Single Node model) or distribute the data across multiple node (Multiple Nodes Model). The distributed data is ready to be process by MapReduce. The different phases of MapReduce are reorganizing the provided data and deliver an output that provide the count of objects for the processed data (Fig. 16).

The above word count example has illustrated the main parts and processes of MapReduce. The input file is text file that has three sentences. The process of MapReduce begins with splitting the sentences in order to Mapp each word with count. The shuffling process is rearranging the words that are similar in same group, in this case we have Hadoop word is mention twice. The reducer process is simply counting the time of each word that is mentioned, in case Hadoop word will become two. The final output file has Hadoop word only mention two times while all other words mention one time.

Based on the above, illustrated word count example has highlighted the functionality and capability of Hadoop and MapReduce.

Hadoop distributed file system provide a method for data analysis with large volume. The setup and preparation of Hadoop in single node model is supporting the analysis on local system, which is cost effective. Another advantage is automated data backup in HDFS, which can provide more security for data. The development of Hadoop is providing more features with each Hadoop release. However, the setup of each version is various and critical in data processes steps. Therefore, this literature review is described the theories and illustrating the setup steps for Apache Hadoop 3.1.1.

The future plane is illustrating the use of Hadoop in Big Data that are generated from social media. Social media is considered as valuable source for many different fields such as marketing and business, science, and technology as well as points of researches for government sectors.

Ethics approval and consent to participate

Not applicable

Consent for publication

Not applicable

Availability of data and materials

Not applicable

Competing interests

Authors are declaring that there are no competing interests

Funding

Not applicable

Authors' contributions

Dr.Youssef Masmoudi : verified, reviewed and edited the content of the paper.
Hatim Almansouri : setup Apache Hadoop, performed the analysis and wrote the manuscript and paper.

Acknowledgements

The authors would like to thank Core Labs at King Abdullah University and Saudi Electronic University and their student Ali Alshahrani and Wael Hakami in their support in part of the literature review.

Authors' information

a. Dr.Youssef Masmoudi

[email protected]

b. Hatim Almansouri

[email protected]

https://www.researchgate.net/profile/Hatim-Almansouri

Afzali, M, Singh, N and Kumar, S 2016, ”Hadoop-MapReduce: A Platform for Mining Large Datasets”. International Conference on Computing for Sustainable Global Development, pp 1856-1859.
Ahlawat, T and Rambola, R 2016, “Literature Review on Big Data”. International Journal of Advancement in Engineering Technology, Management & Applied Science, Vol.5, Issue 5, pp 23-24.
Apache Hadoop. 2018. Apache Hadoop. [ONLINE] Available at: https://hadoop.apache.org. [Accessed 23 November 2018].
Apache Hadoop MapReduce. 2018. MapReduce. [ONLINE] Available at:https://hadoop.apache.org/docs/r3.0.0/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduceTutorial.html. [Accessed 23 November 2018].
Homebrew. 2018. Homebrew. [ONLINE] Available at: https://brew.sh. [Accessed 24 November 2018].
Knoetze, R,Cilliers, C and Wesson, J 2006,”Visual Data Mining of Application Services Data”. Computer Society, Vol.6, pp 1-7.
Lee, G and Raghu, T 2011,”Product Portfolio and Mobile Apps Success: Evidence from App Store Market”. AMCIS 2011 Proceedings-All Submissions, pp 1-5.
Lee, G and Raghu, T 2014,”Determinants of Mobile Apps Success: Evidence from the App Store Market”. Journal of Management Information Systems, Vol.31, pp 133-170.
Nandhini, P,Pavithra, M and Suganya R 2018,”Big Data with Data Mining”. Engineering and Technology, Vol.4, pp 44-50.
Pol, U 2016,”Big Data Analysis Using Hadoop MapReduce”. American Journal of Engineering Research (AJER), Vol.5, pp146-151.
Sivaraman, E and Manickachezian, R 2014,”High Performance and Fault Tolerant Distributed File System fir Big Data Storage and Processing using Hadoop”. ICICA, pp32-38.
Watson, H 2014,”Tutorial:Big Data Analytics: Concepts, Technologies, and Applications”. Communications of the Association for Information Systems, Vol.34, pp 64-70.

No competing interests reported.

Download PDF

Version 1

posted

You are reading this latest preprint version

Hadoop Distributed File System for Big data analysis

Status:

Version 1

Abstract

Figures

I. Introduction

A. Hadoop and Big Data

B. Hadoop Architecture

C. Hadoop prerequisites

D. Hadoop Setup

E. Hadoop Configuration

F. Running Hadoop

Ii. Mapreduce

Iii. Conclusion

Declarations

References

Additional Declarations

Status:

Version 1