MapReduce and Design Patterns - Distinct Pattern Example

In this video we are discussing distinct pattern example

In the previous video we have discussed what is the distinct filtering pattern

So in case of distinct filtering pattern we are not interested to have the repetition of

the same key values but we required to get a set of unique values only

So in this particular video we'll be showing you that how to write the respective Java

course, how to compile it, how to execute it and how to get the output

So distinct pattern example. So here we will use the XML file that is a batches dot XML file and which will find the distinct

batches of the users. So this is the purpose and this assignment in this particular video

So now let us go for one practical demonstration for the easy implementation of this process

So in this video we are discussing that filtering pattern design pattern under this category

we're going to have distinct filters. So we are having the batches.xm. under input batch folder with the size of 21.48 mb

So it is having so many different rows under the batches tag each and every row has got

multiple different attributes. The attributes like your ID, we're having the user ID, name, date, class and also the tag based

So these are the multiple attributes are there under the each and every row

We're supposed to find out all the distinct list. batch names we're supposed to find out so now let me go for the Java code let me go

for the Java code so let me come to this eclipse yes so we are having the class

name that is a distinct batch MR task and here we having two inner classes so one is the mapper class another one is a the reducer class extending so under this mapper class we having the map method and also

we're having this batch titles batch title there is a text under the map method we're having

this XML parsed which is the hash map object getting instantiated with XML to map method

you can see that this is a corresponding method which will take the XML file as input and

returns the hash map object as output and that will be instantiating our XML parsed

so string title is equal to XML parsed or get name so name is the one of the

attributes were there so find the name of the batches so if title is equal to is

equal to null then return need not execute the this part of the code otherwise

batches dot title set title so you are just writing that one you are on these

batches dot title so context dot write ct x is the context object object object

write that is a batches title comma null writable dot get so in this way the

corresponding batches dot that that will be written on to the context this is

my distinct reducer extends reducer so within this reduce method we are

having this context dot write key and null writable dot get so key value pair

will be written there now let us discuss this main function within this

main function we are we know that here we are are going to have the usage will be the distinct batch MR task that is a class name then

input path then output path so two arguments are to be passed so if it is not

so it will exit we have defined one job instance that is a job is the object

name of get instance the name of the job is find the title of the distinct batches we setting the mapper class we setting the reducer class whatever you have defined as the inner classes there

The output key and also the output value. So output key will be of the type of text

That class, output value will be null writable class. So from the argument 0 and from the argument 1, we're just initializing the add input path and set output path, which will be passed as Kamenland argument

depending upon the code value 0 or 1 we can decide that whether the successful

completion took place or not and that value will be returned here now this

next step which will be followed here is to create the respective jar file so

before going for the execution will be creating the respective jar file and for that

purpose we're going to go for the the respective project name then right button

click then export and then jar we're supposed to give the jar path

and the jar file name properly and then next and finish. But already we have created the jar file

So we are not going to execute the process here again. Just going for right button click the export, then jar files and rest of the steps

Let me execute our command here. So let me clear the console

We're going to write the command for execution. Haddub, then jar. Then we are going to give the jar file path

here that was so slash then we should go for map produce underscore design pattern then

slash jar file so these are respective path and filtering pattern or jar is a jar file

name then this is our distinct uh distinct filter is a respective package name class name is our distinct batch immer task then

you shall give the input path name so that is our slash input slash batch then

the output path name that is a slash output so in this way the command will be

executed so we require to find out where we require to filter out all the

distinct batch names so that is about purpose in the batches dot XML we had one attribute

name so you see the command has got executed successfully so let me see that

what the part file has got created so we shall go for this root of the name

node then output folder we're having this part file has got created containing the

contents so let me go for the command to see the contents here so is DFS

DFS slash minus CAT then slash output slash part star so in this way the part file content

is being displayed so you can find that these are the distinct batch names see these are

the distinct batch names we have we have obtained in the output file these are the

distinct batch names you can easily find that what I mean so many distinct batch names

are there they have got selected in the output file here in the part file under the output folder so now it is a good practice to delete

the output folder so that we can execute the next map reduce task so we are going to

delete that one I hope that you have got the conception that how to execute this

this program using the proper steps and all we have displayed thanks for watching