MapReduce and Design Patterns - Shuffling Pattern Example

In this video we are discussing Suffling pattern example

So in this example, we'll be implementing suffling design pattern. So, shuffling pattern example, it takes the comments.xml and randomize the records using

suffling design pattern. So here we'll be taking this comments. . XML, that is the XML file, where the datasets are deciding, and we'll be doing the

suffling design pattern implementation on this XML. So let us go for one practical demonstration

to show you that how the Java code can be written and how it can be executed and how the

outputs can be obtained. Here we are discussing one problem that is a shuffling pattern

example under data organization design pattern. We are having the comments.xml under the folder

that is slash input slash comments with the size of 37.98. mb so this comments is containing so many different comments by the user so here we have selected

some of the rows of the comment within the comments tag we're having multiple rows are there

so this is a one row now each and every row has got multiple fields like id post ID is code

then text then creation date and the last one is the user ID so we're having multiple

such rows are there we shall be going for shuffling all these roles in a random order so

here that only only one Java class is there that is a common shuffle MR task

so which is which is having one inner class that is about shuffle comment comments mapper

which is extending the mapper we have created in writable object random object and also

we are defined another one there is a random object is there and then we are going to define another one there is a text output value of the type of text

We're just overriding the method that is a map. Under this map XML parts is a hash map key object which will be instantiated by XML to map this particular function will take one XML file as input and returns the hash map object as output

So that function has been called here. So we'll be going for XML parts dot size if it is greater than zero

then only will be going for the true part. So we are going to define one string builder object That is a string builder is a new string builder string builder We having this a string builder object and it is going for append that is a less than

row one blank space just check here we're just going to append this one less than row

and blank space this string will be appended. Now we shall we shall check so here you can find that this a row actually so we

shall create such rows in my in our output file we shall be creating such rules in our output file so that I'm showing you here okay now the

next thing is that for entry string string entry within the XML parts dot entry

set so for this XML parts containing the hash map key if the entry dot get key

dot equals user ID or entry dot get key dot equals ID any one of them then go for

ignored don't consider that one we are not going to write that one on to our string builder class object else if entry dot get key equals creation date

in that case what we're doing and string builder dot append entry dot get key

but just writing the key there then we are putting one one double

quotation so that's why you written that one that is a that is a double

quote is equal to slash double quote then entry dot get value so one double code you are putting then entry dot get value dot sub string

0 to entry dot get value dot index of capital t now what is this so you are

taking a sub string till the capital t now question is coming in mind why this

capital t is residing so let me go back to my XML file so that I can show you the

date time format just come to this XML file you see just go for the creation date

you can just see at a creation date but having this one is the date and then capital t is there

which is denoting the time and then the time stamp is coming so this capital t is actually denoting

a delimited between the date and the time so until i get this capital t so i'll be

searching this capital t with the help of index so until that one from the very first

index to the capital d index prior to that that is a sub-string so that we are doing here

and then we'll be going for slash. So we one slash double code so double code will be appended Otherwise string builder dot append the entry dot key and then slash one double quote is there and then we be having the entry dot get value the slash double quote

So you see we have given the blank space we're given the equal to sign everything we have done and at the end we're just closing that angular bracket

So just remember this one in the output will be reflecting all this

So output key dot set that is a random. Next I and output value dot set string builder dot two string

2 string so the output key and output value has been initialized and now we'll be

writing onto the context the key value pair in the context will be writing that

one enclosing the code in the proper try-catch block including the code in the

proper try-catch block now we are going for the shuffle comments reducer extends

reducer here we are going to override the reduce method as we usually do you see

in the reducer we're going for this text value working on the values

and values is nothing but one iterable class object now we are going for for loop text

value which will be working on the values values is iterable class object and writing the key

value pair there so value and null writable dot get so now we are going to discuss the main

in the main you can find that we require the class name and then input folder and the output

folder input folder so two arguments are to be passed if you don't if we didn't then obviously

error will be there so creating one job that is a shuffle the comments setting the

jar class setting the mapper class and reducer class already you have

defined this mapper class and reduce a class earlier so here is the mapper

class and here is the deducer class we have defined that one earlier also we have

set numb reduced tax two so to set two reducers are there

Text input format will be that set input paths will be arc zero and set output path will

be arcs 1 and output key class will be intratable output hello class will be

text class we are going to check the completion status zero or three in this way

the program would be executed so now we are going to create the jar file so

package name then right click export jar respective path and the jar file name has to be

given and then next next finish but we have already created the jar files

were skipping this particular step so let me execute the command for the execution of this so hadup jar then the respective path

we have produced design patterns slash jar file slash there is a class there is a

class there is a jar file name and then we'll be going for

there is a suffling pattern is a package name and then comment

shuffle is the shuffle immer task there is a class name then you shall go

for slash input so comment so this is a input so comment so this is a input

path and the output path so we shall execute the command now so here we'll be

suffering and you can recall that how we have planned to write the rows that is a

there is a angular bracket and then blank space then row we'll be writing

the the syntax we'll be writing only the date no time part so let me let me show you the content so s dfs dfs minus cat

then you'll be going for here the output path and then part files will be there part star

see the contents see each and every row which whichever has come onto the part files we have written

because we had the string be glass object so we are having this just taken in

you can find that we are having that angular bracket row blank space creation

dates code this and that you see in the date we're having only the date part no time part but having only the date part and rest we wrote accordingly and then

a closing angular bracket so in this way you have you have seen that how this

this corresponding content has been suffered into our output file I think we

have explained the code line by line let me delete the output folder as we do

earlier also so I hope you have in enjoyed this video and thanks for watching