MapReduce and Design Patterns - Partitioning Pattern Example

In this video we are discussing partitioning pattern example

So here this particular video will be associated with one practical demonstration for the

easy understanding of the implementation. So in this example we shall be using the postlinks.xml to divide the post links based on

their respective criteria. So there are two different post links in the dataset, the post links with the value

1 and the post links with the value 3. So let us go for one practical demonstration of the implementation of this problem and to know

it better. So here is the demonstration for you. In this example we are going to discuss another data organization pattern

That is a partition pattern example. We are having the XML file that is a postlinks.xml and we shall partition this post links into

the type of ID 1 and ID 3. So we are supposed to make partitions we'll be creating two part files in the output that is with the ID1 and ID3

So how to execute this one? So we're having slash input slash post links is the path in which the file is existing with the size 2.35 mb

So here is the postlinks.xml. We have kept some of the rows but there are so many rows are there

Under the post links tags we're having each and every row consisting of ID, then creation date, post ID and related

post ID and the link type ID so here we're having one and some of the all we'll be

having some other values than one we're going for this Java program only one Java

program that is a part post link AMR task so here we're having multiple inner

classes the first inner class is our links partitioner this inner class is

extending the partitioner class and within that we're defining one's private static final string that is a partition values is equal to partitioned values so

there is a final string we are creating here and also the other instance

variables like your configuration config string partitions and add a list part list

of the type which will be holding of the type of data string and then these are

the different instance variables will be reference variable will be used here

we're going to override the method that is a configuration get confit conf which is

returning the configuration here this is a get conf also we are overriding the

method that is a set confine set configuration and the conf here we are going for the set conf here so this conf you a confe is equal to conf what about the input argument partitions is equal to conch dot get

partition values so partition values is the respective constant string which you

declared earlier and now these partitions will be splitted these are partitions

values which you define the final string now these partitions will be

splated taking the blank space as a delimiter into this particular list at a

list you can find that one the part splits so these are delimited is a blank space

now we shall go and t that one so that we should not want to have any kind of

blank spaces prevented and appended extra so after doing the t we are just

adding it to the part list which you declared earlier so in this way we'll just use

the blank space as the respective delimiter splitting it t it and then adding it to

the part list I think you are getting my point so next one

you are going to override the gate partition so get partition is another method

within the partition class so here we're having the string p value is equal to key

dot two string dot trim so from the key whatever has been passed as in writable

so this p value is getting initialized and the t after t if part

list dot contains p value then return part list dot index of p value so the respective

index of the p value will be returned else it will return the numb part

that is the I&T which has been passed as input argument so it will return this

num partitions so else it will return the numb partitions so that we passed as

input argument next we're defining one other method that is a set partitions

so static void set partitions which will take the job job and string

st here so job dot get configuration dot set parameter values comma string

styr so this is the respective the job configuration is getting set here

Now we are having one mapper class the name of the mapper class is post links mapper

which is extending the mapper having got some member variables here so we are having

this int writable output key and also the map we are overrating the map method

so XML parts you know we're having one function one function is about XML to map

which will take one XML as input and returns the hash map object as output and

that hash map will instantiate XML XML X parts that is a hash map object here we're going for the string link type is

equal to XML parts dot get so link type ID so from that very link red ID from

the XML that is a hash map we initializing this link type if link type is equal to null then you shall go for return otherwise link type is equal to integer dot value of link type dot I nt value so converting it to the integer we are

updating that one and output key will be updated with this INT link already

we have defined this output key that is an int writable so now we are going to

write the benefits dot right that is a key and value paired OPE key is containing

the key and the value is the value and that is getting written temporarily

We kept it under this try catch block. Here we're having the reducer name of the class is post links reducer

So this is a reducer class which is just we're overriding the reduce method

Text values dot values. So values is iterable there. So context. .write value comma null writable

So here in the context we're writing the key value pair here

So now let me discuss this main function. this main function requires three common land arguments if the length is not

equal to three that is error three arguments are input and this is a string that

is one blank space three I told you that we're having this type ID will be

one and three and also we required the output output file output folder defining

the job which is the job will be name has partition the post links by link

type ID that is a job we are defining also where a the jurr class and then the mapper class already we defined the partitional class

we defined set partitions that is a job comma argument one so argument one means

what that is a that is a word there is a one and three so there is argument one

and three yes now go for job dot set reducer we're just rid of putting the reducer

there whatever you have defined already we have just putting that one and then

we're going to create two reducers two partitions you know where going to have two partitions one for partition with the type ID one one for another

one for type ID 3 so we require two reducers two part files will be there in

the output so we are going for add input path and set output path with the

argument zero and argument two input path and output path so that is the input

and the argument two will be the output path whatever you are giving then we are

going for set output key class set output value class set output format class so everything we defined earlier's we're just mentioning that

one and depending upon the completion either zero or one will be returned so now

we shall create the jar file so we shall go for the respective project and then go for the right button click as we show we have shown that one for multiple times then export and select the jar file name and the respective path click on finish and next and

finish in this way the jar file has got created we have already created the jar file so you are going to execute the command so Hadoop jar then we're having

this map produced design pattern jar files is a folder and data organized

patterned or jar is the jar file name then after that we're giving the package name that is a partition pattern

is a package name and we're having the class there input folder we're giving that is a post

links and this is the string we're passing that is one and three and the output folder so as we

mentioned earlier so these are the respective parameters we are passing we're executing

the command I hope that we'll be getting two part files one for ID one another for ID 3 so let me go for the output folder

so here is the output folder there you can find that we are having this part

file ending with zero another part file ending with one so two part files are

there one for ID 1 another 4 ID 3 so let me

agree see the content so going back back to the console again

going back to the console and we are going to print the part file content

so here we had deducer to so the command will be HDFS dfs minus cat slash

will be going for say slash output slash the file name is part and then are then

zeros and then all zeros will be there so five zeros and then press center

you can find that in this case just look at the last one there is a link type ID is

for all the cases so that is one partition we have done we've done this data

organization so one partition we have done see the last field link type ID is

equal to one otherwise we're having the ID creation data and other things we're

having for the next one link type ID is equal to three here so in this way

you have you have seen that how we have implemented this program we have discussed

line by line step by step and so on we can delete the output folder as usual

for the next MR tucks to execute. I think you got my point. This is the command I'm issuing

And this output folder will be deleted. Thanks for watching this video