MapReduce and Design Patterns - Average MapReduce

In this video we are discussing average map reduce

Now what is average map produce? So in this problem we will use comments.xml

So we are having another XML there. There is a comments.comels.coml. And this comments dot XML file will be used in this average map reduce

Using comments, we will count the number of comments posted on each user for each user and

also find the average length of their comments. So that means it will count the number of comments made by each user and the average length

of the comment. So that will get calculated here. So here also the key is the user ID and the values are average comment length

So that will be the output will be going to get. So let us go for one practical demonstration for the easy understanding of this concept

So here we are implementing the average count MR task which is falling under the

summarization design pattern comments dot XML under the folder slash input slash comments so under this folder this comments dot XML is a file which is our

input file so let us see that what is the current content of this comments dot XML

we're having a huge content it is XML file so we are having a huge content but here

we have shown a part of that so under the comment tags we're having the

row tags and row is having ID the post ID the score and the respective comment whatever he or she has made the respective creation date and also the user ID

so these are the fields are there under the row tag in this way we're having several such row tags are there under this comment

under this comments tag so this is our contents of comments.xml so here we are going to have our two

Java programs one is the average count data. Java and another one is the average count MR task

Java so in case of average count data implements the writable implements the

writeable interface it is having the constructor and there is the average count data

it is having some getter and setter methods and two methods are also there

one is a read fields and another one is the right fields the member variables are

here the common count and the average length so these are to the member variables and within the constructor we have initialized them these are the respective gator and setter methods that is a get comment count get average length set comment count set average length

We're having the read fields right and also we have overwritten the method that is two string

So this average length you are finding here we are just initializing that one in the set average length

In the average count MR task which is the main Java file here, which is having a very

is having the respective mapper which is having one inner class that is the average

count mapper which is implement or that inheriting the mapper class and also

we're having the reducer class has got inherited by average count reducer so within

this average count mapper we're having the text type of that is OP user ID and

average count data objects are there two member variables and also we have

overriding the map method XML to map is one method we have written which will be

converting this XML content to the hash map type so it is calculating the x a

XML key and so on so that XML to map the method whatever will be returned

will be kept in XML parsed and XML parts dot get user ID XML parts dot get

text that will be kept in user and comment text so within the tri-block we are

just initialize this set comment count as one set average length will be the

comment text dot length so initially they it will be initialized and then op user

ID dot set user and then we're writing the context that is the op user ID and

the OV OP average count value so these two values will be written as a key value

pair on the context so that it has been kept within the tri-block in the

reducer we're having this this reducer class has been inherited by average count reducer which is overriding the method

reduce here which is overriding the method reduce we're having the common

count is equal to zero and common length sum is equal to zero also so in the

average count data for each average count data into values we're just writing

common count plus equal to data dot get common count so common count will get

increased and also the count length will be increased by the data dot get average length into the data dot comment count so comment count and the average length will be multiplied and that will the product does obtain will increase the count

length sum and then we are just setting we are using the set methods that is a set

comment count and set average length so with these values will be going for the set

and there is a count length sum by comment count and that we are going to write as a key

value pair onto the context so also we have discussed this XML to map already we have discussed so and you know that in

this way we are having the main method here the main method requires two

arguments so if two arguments have not been passed so it will cause some

error and it will exit system exits two and then if it first argument will be

accessed as a are g s zero and the second argument will be accessed as air g s 1

so that will initialize the that will take the file input format and the file output

so that will be kept and the set mapper and the set reducer class will be

initialized accordingly whatever you have discussed whatever you have discussed earlier that is inheriting the mapper and the reducer class and then we'll

be going for job dot set output key and then job dot set output a value class so

these two values will be updated and then depending upon the success the exit

zero or exit one will take place so it will it will be the bullion one there so

now let me go for the execution of these two Java classes so before going we

shall create the jar file so how to create just on this particular package name

go for right button click and then go for export and then jar then we shall be

giving the proper name and the folder and then you shall go for next and finish

but already we have created the jar so I'm not going to finish that steps but these

are the steps to be followed let me show that how the command has to be

issued so Hadoop jar that is a map produce design pattern jar files is a

respective folder and then summarization patterns dot jar and then we are having

the average count that is a package name and then average count MR tax is

the respective class name and then under the input comments within this folder

we'll be having the input file that is comments dot XML and slash output will be the

output folder on in which the part files will be will be will be created after the running of this comment so let us run this comment at first I not finding the success

the name node is name node server is now in the safe state

so let me make it come out from the safe state so will be for the name node will

be I'm going to retrieve the name note getting out from the safe state so the

command will be how do there is a DFS admin then minus safe mode then leave so

now I think it is okay now let me execute the command once again so if I get

success then under the output folder the parts file will be created which will be containing the respective output yes we

have got the output file created so let me check here so comments dot XML is input

input folder was there and now we are going for the output folder and

there we're getting this part hyphen r hyphen zero so here is a part file only a

single part file has got created so here the output has been stored so let me go

for the console back again and let me go for the sdf sdf s command with minus cat option to

see this to look at this part file content so slash output is a path name and then part

star so all the files starting with the part its content will be printed so there is

the respective content of the part file we're having the ID then we're having

the comment count and how many comments he or she has made and then we'll be

having the average comment length so for 4 to 2.5 is a comment length so in

this way you are finding a rest of the files content rest of the content of the part

file so let me delete this output folder because the other map produce programs

will be accessing the same folder there so let me delete that one it is not mandatory but for the next execution

So, these are delete comment