MapReduce and Design Patterns - Download and Setup Sample data into HDFS

Download and set up sample data into our SDFS

So we require some sample data for running our MapReduce programs. So how to download them, how to keep them in the SDFS

We'll be discussing that one step by step. So we shall download the sample data to run our MapReduce programs

And we shall use the stack exchange data dump. And you can get all the download links here

So this is a download link. From here you are supposed to download this file that is Android.dot stackexchange.com

So this file has to be downloaded. It is containing different datasets

Now set up HDFS and store downloaded data and we should create some directory in HDFS

and then store the respective XML files into it. In our example, we have created slash

input folder and then another folders to store the respective XML so on in the

SDFS route will be creating one folder the name of the folder will be input

under that folder folder will be creating multiple subfolders and the respective XML files will be stored on them so under this particular input

folder we're having this badge folder and then batch. XML we shall create

another folder comments comments.xm. Post posts dot XML in this way you can find

that all these 1, 2, 3, 4, 5, 6, 7, 8 XML files will be saved or will be transferred to this

particular path and they'll be used in our MapReduce Applications Development. So let us go for one demonstration for the easy understanding of this concept

At first we shall copy one URL here So here is the URL that is the HTTP colon slash slash archive slash download slash stack exchange So we shall copy this URL and we shall paste it onto this browser URL box and paste and go

This webpage has got open and here we are going to select Android

stack exchange.com.7.Z. So this particular file is to be downloaded. But actually I have

already downloaded this one on my download folder. So I'm not going for okay. I'm going for

cancel. Otherwise I shall have to download that one. So I shall open the download folder. The file is

there. So we shall go for the right click and we shall go for extract here. A new folder has got

created Android.com.com. So the extraxact. instruction is taking place

Extruction has got completed. So we are opening the folder. This folder is having 8 XML files

You can find that. It is having 8 XML files. So we shall copy all these files onto in the folder that is a map produced design pattern

So we have gone to the map produce design pattern going for a new folder creation

Let the folder name is XML underscore files. We are creating a new folder under the MapReduce Design Pattern folder

So we shall go for the respective download folder and there we are going to copy all

the 8 XML files, going for the Home, then snakes, then MapReduce Design Pattern

then XML files. Now we are going to paste all the 8 XML files here

So now these XML files have to be put in the HDFS so Hadoop has to be started

we're opening one terminal so we are going for the Hadoop start now Hadoop is

starting we shall also issue the command JPS to say whether Hadoop processes are running or not JPS all the processes are running

all the processes are running now we shall open the browser and there is a

Hadoop browser is there we are going for the refresh there is one folder called

input in the sdfs route this input folder is void it is blank now we shall

create so I'm not going to execute that one that is mkdir for input folder so I'm

going to create eight different folders under input folder and there will

be keeping the respective XML files so the first folder has got created there is

a slash input slash batch now going for the rest seven folders creation on HDFS

And then we shall copy all the eight XML files to the respective folders

We are supposed to create another four folders

You know We shall copy XML files there respectively after creation of all 8 folders

So all these files are to be copied

So this is the there is a command there so put command so copying files from the local system to the SDFS

so let me copy all the 8 XML files in the respective folders

So in this way we shall create all the folders onto the SDFS and then we shall

copy all these XML files to the respective folders and now let us check the input

folder now and you can find that all the XMLs have got copied