MapReduce and Design Patterns - Min Max Count MapReduce

In this video we are discussing mean max count map reduce

So this particular problem we shall go on implementation also and we'll be working one XML containing

such data. So let us discuss on this problem into more details

So what is mean max count map reduce? So in the map produce task we will group the batches.xml data on the user ID

So we are having one XML that is a batches dot XML. So they are existing in the respective folder

We have shown that one in the previous videos. And then on the user ID, there is one column will be there known as the user ID will be forming group over them

And for each user, we will find the minimum and the maximum the daytime of getting a batch

So against each and every user ID, we shall find out the minimum and the maximum date time of getting a batch

So also count the number of batches of that user. So these are the tasks we are going to do

So that's why it is called mean max and count map reduce

So in this example, the user ID is the key and mean underscore date, max underscore date and counts are the respective values

So let us go for one practical demonstration of easy implementation of this problem

We are discussing mean max count problem in summarization design pattern. we're having this file system here we're having the file that is a batches dot

XML batches dot XML which is our input file and we'll be discussing on this batches dot

XML of but at first we want to concentrate to discuss about the Java files so

batches dot XML is about input file which is under the folder input slash batch so

under this folder the batches dot XML is there so we are opening a clips here

we are having two Java classes are there one is a mean max sum data dot Java and another

one is a mean max sum m r tux dot Java so we are discussing mean max sum data

or Java it is implementing the writeable interface and it is having the member

variables mean date max date and sum some some means actually here we'll be

calculating the count here we're having the constructor that is a mean max sum data

is our respective constructor here and we're having the other methods are also

there in this particular class we're having this is the date format this a

format of the date in which the date will be interpreted and we're having

get a mean date get max date and get some so these are the geter methods and

get mean date set mean date set max date and set sum read fields right and two

string are the other methods are there so read fields right andpack

string and some getter and setter methods and also the respective constructor we are

discussing mean max sum m r tux dot java so there is another Java class which is

extending mapper a mapper class which is inheriting the mapper class and here we having the method that is a map we having the method that is a map and also the other methods are there will be discussing having the respective methods like our say mean max sum

mapper which is extending the mapper class here and then also this is the

opi data text op user ID and mean max sum data op data so this data is there so these are the variables are there will be

using and also we are going to have one method that is the XML to map which

converts the XML to hash map so let us discuss this particular method at first

so there is our XML to map which takes a string as input argument and which

returns one hash map so this is a conversion method and that has been

called here to which whose output will be kept in XML parsed so XML parsed dot get debt and XML parsed dot get

user ID so this is the user that is a string user and also the string

creation date so these two variables will be initialized using this XML parsed

dot get methods we're having this date format the already the date format is we

have discussed the date format so this is the date format in which we'll be expecting our debts in our

data file you can find that these are the respective setter methods are there

which has been kept in the tri-block and catch block is there which is holding

the catch block to catch the other exceptions if there is any exception has

occurred this a count dot right that is a user ID and the opi data so we'll

be counting that how many times this particular paired has occurred and this is a

our mean max sum reducer which is extending the reducer class initializing this is a

reducer this is a reducer class which has been inherited here and we have

overwritten the method reduce in this particular in this particular section obdata dot set max data null obdata dot set mean data null and obdata dot set sum that is

the initial count for the for the for this particular value we have kept that one

zero at first and final sum has also been initialized with zero so if mean max sum

data so now here we're having to if conditions so if obdata dot get mean

date is equal to null or obdata dot get mean date dot compares to get mean

mean date so depending upon this condition so we are going to update the mean

date with the data dot get mean date otherwise if we compare for the max date also

So if the max date is not is equal to null or say obd

.get max date. . compared to this is less than zero, then the op data

. . . . max data, max date will be initialized with data . . . . . . . max date

So in this way the set max date and set mean debt methods has been called from the op data object to keep the mean date and max debt respectively depending upon the conditions and also the final sum has got increased by the current count so final sum plus equal to data dot get sum so

add and sum with the previous sum and here we are writing this context we are

writing this context so ctx dot write we have used this method there there is a map

this a map method and let us come to the main method

now right at this moment so here we require two arguments to be passed so if the number

of arguments is is not two in that case will be exiting with some error message

printing now for the first argument will be accessed with argument arg zero and the second

argument will be accessed as arcs one because we require two arguments to be passed here

so job job is equal to job dot get instance config mean max some job and the rest

respective one and then job dot set jr by class mean max sum mr task dot class so that

class whatever you have defined we've mentioned that one we require two arguments the

first one will be the input so add input path and the second argument that is the arc

zero will be the output so set output path here so that will be passing in the command

and argument whenever will be running the code set mapper class that is a mean max sum

mapper and set reducer that is a mean max set reducer class so these two classes where

just assigning to this set mapper and set reducer classes so also we're having

the bullion so if the success is there then the exit will be done with zero otherwise exit

will be done till one so we shall now go for the the batches dot XML this is a batches dot XML

you can find that we're having the multiple rows are there we have taken a snapshot of that so here

it is one XML files we're having the ID that is ID is 1 2 3 4 in this way user ID is also

1 3 5 8 10 and so on so we are having different kinds of types of users we're

having autobiographer we can have the teacher we can have student we can have

supporter that are so many different category of users and also we're having the

respective date and time is there in the same format in which we expected there we're

having the class IDs and the tag based one Boolean field is there in our code we

didn't use the tag based but also we can use that one so it is under the folder

that is a slash input slash batch and the file name is batches dot XML so we have

got this Java files we have got the idea of this batches dot XML that is the

input file now we shall show you that how to run the code we have opened the

terminal let us clear it now we shall write the command so it will be a

the Hadoop jar there is a path map reduce destination pattern slash jar file

summarization patterns dot jar this jar file we shall be creating from here see we'll be

going for the summarization pair problems and then go for the right button click and then export and then you shall export it to the jar file is the jar file is the respective folder in which the summarization problem will be will be kept as

a jar file in the in the Java archive resource you know there is a Java

archive so in this way the jar files will be created now let us let us come back to

the command prompt so this is the jar file and the respective path has been

provided and input will be about uh... means min max sum dot min max sum mr task so that is a respective input class here there is a class

actually which will be executed then we require the input file input file is under the

folder slash input slash batch so under this folder the input file will be existing

and then we'll be also having the output there will be also having the

respective output output is a folder in which the output will be created now we shall

execute this command so outputs are to be created under the output folder in the

form of a part so output has been created in the output folder in the form of a part

part file so let me open that one so these are batches dot XML that is the input

and here we are having one output folder if you open the output folder we are having

the part hyphen r hyphen zeros are there so there is a output file so we are supposed to see the

content of this output file here so let me go for the output file let me go for the terminal

and then we shall go for the cat command through sdf s

we're giving the path name and then any file starting with the part need not to write

the full file name so any file starting with the part so that will be printed

because the command the option we have given as cat so the content will be

printed yes this is the content so you can find that in the first column

we're having the IDs then we're having the mean date we're having the max date

and the number of batches given to the respective ID. So here the number count is about 29

So there is a count actually the last column is containing the count. ID, mean date, max date and the count

So that was our purpose to print. So that has got printed in the output folder

So now to a we'll be running so many other files. So we can also delete this output folder

You can find that we can we shall be deleting the output folder. let me issue the comment here so we shall be deleting the output folder so that

next time I did not to face any problem it is not mandatory so minus RM for

remove and recursive so the file folder has been deleted thanks for watching this video