MapReduce and Design Patterns - Structured to Hierarchical Pattern Example

In this video, we are discussing Structure to Hierarchical Pattern Example

So, this video will be assisted one practical demonstration for the easy understanding of this concept

So, this task takes the Posts.xml and also the Comments.xml to create a hierarchical dataset

So here we are using two XML files. One is the Posts.xml and Comments.coml

XML and they will be considered and then the output will be in the hierarchical

data set format so let us go for the practical demonstration for the easy

understanding of this under data organization pattern we are going to discuss

structured to hierarchical pattern example in this particular example we are having two XML files users dot XML under the path slash input slash user and

comments dot XML under the path slash input slash comment and we are finding out that which user has made which comment on certain post so

slash input slash user we're having users dot XML so this XML file is containing

user details and if you go for slash input slash comments comments dot XML here

this XML file is containing comment details so let us show the current content of

this there are so many records are there but we have just taken some of them for

your understanding at first we're discussing users dot XML within the user's tag

we're having multiple row tags are there with multiple attributes so attributes

are something like your here for each and be row ID reputation we're having

the creation date display name last access date website URL we're going to

have location about me views upvotes down votes account ID so these are the attributes are there under the

row in case of comments dot XML we're having the ID post ID score text creation date

and also the user ID under each and every row and whole thing is under the comments

tag so we have discussed what is the users dot XML and comments dot XML let me go

for the Java program here we're having only one class that is user comment

higher m r tux dot java it is having the inner class that is our user comment mapper which extends

mapper so user comments mapper is the inner class which extends mapper and also we are having

another reducer class here so within that we are having two variables one is the there is the output

key which is of the type of text and output value which is of the type of text so output key and

output value we are overriding the map method we're overheading the map method we're overheading the map

method here so here we having the XML pass that is a hash map object and we are having XML to map one method that method will take one XML as input and returns the hash map object as output which will be instantiating this XML

parts but defining one variable that is a ID here and from the XML parts from

the hash map we are trying to get the ID that is a respective tag so ID we are

putting the value in string ID so if ID is equal to is equal to null then return

we are not going for the rest part otherwise the output key will be set with xml parts

dot get ID so the respective ID value will be put in the output key and then the output

value will be set with the ID value but along with that u plus conceding u as a

first letter with this ID and we're writing that one temporarily on the context

that is the output key and output value so key value pair will be written on the

context dot right so user for user this respective you will be the first character for user the output

value will have the first character that is you capital u in case of comment

mapper which is also extending the mapper class we're having two variables

there that is the output key and output value both of the type text again we're

having the XML parts that is a hash map object and XML to map is instantiating

that one so here we're taking this user ID from the hash map we are taking this user ID and that is getting assigned onto the

string user ID if user ID is equal to null then we're returning the controls

so this part will remain unexecuted output key dot set XML parse dot get owner user ID

so that will be kept as a key and in case of value here we are conceding with

the capital C as a first letter and then value to string to be converted to string

and that will be kept so add letter C with the comment

So there we had the user with the later capital you here with the capital C

We kept everything within the tri-catch block. But having the reduce, that is the way are having the reduce, reducer class has been extended

We're having one user ID and the comment list. There is an array list there

User ID has been initialized with null and comment list which is the array list has been made clear at first

Now we shall go for this this text value values is nothing but one iterative

object you can find that one will be taking one at a time so if the value dot

carrot is zero that means if the first character when a user ID is found that

we remove the letter you and take the ID so we'll be taking the substring from

position number one to the last during the t and that will initialize

this user ID and in case of in case of comment the first character is C so

comment dot list dot add we are taking the first character onwards how fast

character at the character at the 0th index will be discarded and we'll be doing the

t and that will that will be added with a comment list So if user ID is not equal to now then we are creating one JSON object and we are putting that one user user ID into

the JSON object and in case of JSON array we are creating one instance of JSON

array and JSON array dot put comment list and JSON object dot put comments JSON array and

then we are again writing onto the context dot write there is a new text

Jason object to string and null writable.get. So there is a key value will be written on the context

And here you also have put within the try catch block. So here is the main function

So just see the usage here. There is a user comment hierarchical MR tax. So that is a class name and then user input

then comment input the respective to folders user input folder where the XML will

be deciding comment input folder whether comment will be deciding and also the

output folder so these three parameters are to be passed so if the argument

counts is not equal to three that means the wrong argument so we shall not go otherwise we shall go for creating a job instance and job name is get user ID and

comment in JSON format so job dot set jar by class where mentioned

the respective class here we're having the multiple inputs so add input path

job comma there is an argument zero that is a first argument whatever you are passing

a text input format class and user comment mapper class so these two class we are

mentioning for the multiple inputs will be taken there and then multiple inputs will be going

for add input path for the next now going for the set reducer class so reducer class is

user comment reducer class which you defined earlier so there is arc 0 and arcs 1 will initialize the respective add input path in both cases

because i'm just marking that one to show you that what are the classes we're using that one

then we'll be going for job dot set output format we'll be going for job dot set output format class

there is our text output format class and text text output format dot set output path

will be job and new path argument too. So that is a third argument, whatever you're passing

there is an output path we are passing. So job. dot set output key, there is a output key class is text

dot class and also output value class will be text. Dot class

Depending upon the system exit, the current condition of the completion, that value will be returned

either 0 or 1, 0 means success, 1 means failure. Now after creating this Java file

we shall go for the jar file creation. So we'll be going for the package

and then we shall go for export then we shall go for the jar file we shall click on the next we shall find out the jar file part and the jar file name then we shall go for next and finish but already we have created such jar file earlier so we are not going to create it once again I just keeping this particular step here so let me go for the command so how do

jar we're writing hadup then jar then map produce design pattern so there is a

path jar files is a path and then we're having the data organized pattern

dot jar so there is a jar file name we mentioned then struct to hire is the

respective package name then user com hired dot m r tux is a class name so this

a package and rest one is a class I'm just marking that one for the better

visualization then input path will be the where the user XML will be there

users XML there is a slash input slash user and then input XML input XML that is a

the comments XML so that will be under slash input slash comment and output will be the folder

I'm executing the command see the outputs it will create the part files and I shall show show you

the display of the part file content against user ID the comments will be displayed if the comments

are there so it will display the user ID and the respective comment in the form of a list in the

part file so part file will be created under the output folder so let me go for the

cat of that folder that means the content of the folder will be going to display

so HDFS dFS minus cat and then slash output slash part star so here you can find

that we're having the list of all user IDs and the respective comments are

there all user IDs and respective comments where the comments are not there it

is showing the blank and whenever comments are there it will be showing the respective comment you can find that so many user IDs and the respective

comments he or she has made on multiple different posts so this is the actual

the basic theme behind the structure to hierarchical pattern example we have shown

that one into detail step by step so that you can follow we can pause the video

and we can get it done so it is our common practice to also to show you that

this is the user ID and the respective comments will be printed. So each and every row will be user ID and the respect to comments

Most of the comments are null, some of the comments are also existing

So let us go for the deletion of the output folder and then you shall conclude

So the command which will be going to execute is that because the deletion of the output

command output folder is a common practice in this way. We'll be going for DFS, SDFS, DFS minus R minus R, slash output

slash yes that is okay now the output folder has got deleted thanks for watching