MapReduce and Design Patterns - Binning Pattern Example

In this video we are discussing binning pattern example and using this example we shall also give

you some idea regarding the implementation of the code and running of the code

So in this example we shall be using postlinks.xml to divide the post links based on their

post links and the code will generate two files, one is the duplicate another one is the link type

links from the XML file. So to have this particular assignment getting implemented, so let us go

for one practical demonstration for the easy understanding of the concept. In this example

we are going to discuss binning pattern example falling under the data organization pattern

We shall be dealing with the postlinks.xml under the folder slash input slash post links. And here we're

We are going to discuss the current content of this post links.xmlちょっと we have shown some of the rows, all the rows have got enclosed within this post link, having got multiple rows are there and these are the I, the respective attributes I'm showing here

So ID creation date, post ID, related post ID and linked type ID. So these are the multiple attributes are there

So here we shall separate links by link type ID and it will be only a map only job. So we're having multiple rows are there. Here we have shown some of

them to make you understand this is my Java program so binning mrt tasks. Java

under that I'm having one mapper class extended that is a links mapper under

this we have defined one variable one object that is a multiple op that means

output under the class multiple outputs and we are having the setup and within the setup we shall in instantiate this multiple output variable This multiple output is equal to new multiple outputs in the current context so that has been done in the setup method

we have done the overriding next one we are going to override the map method this map method

we're having one xml parts that is the there is a hash map there is a hash map object

and xml to map is a method which will read one xml file and returns one hash map object

and that will instantiate this XML parsed. So now string link type is equal to XML parsed

Get link type ID. So using that ID from the hash map, you are getting the link type

If the link type is equal to is equal to null, then return means rest of the code will not be executed

But here this link type can be having the value either one or three

So if the link type dot trim, thus eliminating the blank spaces before after, if equals one

then you shall go for this multiple output right, bin. value null writable dot gate and link so this particular text is there so separate

the links for the link tab ID is equal to one and then in the else part we are

going for the same but here we're writing that one for duplicate we have

kept that one in the tricatch block we're having the cleanup where we shall only

close the multiple output object we're having the respective let us discuss it is a

it is a new reducer only it is having the map only job so we shall discuss the main function this main function with the

request when the program will be executed two arguments two arguments to be

required the first one is a bidding MR task that is a class name then the

input file location the folder and the output file location folder so that is

required otherwise exit one job has been defined with the name that is a

binning the post links so that is a job ID so there is a job name we have defined we have assigned the respective job where We have assigned the respective mapper class whatever you have defined earlier And here the reducer there is no reducer map only job

And for this set input paths and set output path will be using this argument 01, argument 1, common line arguments

And therefore, multiple outputs. We are going for add named output. There is a job, bins

There is a bins. Then we'll be having the text format which is of the text format class and then text class and output will be null writable class

Next one we're having the multiple outputs or set counter enable so job true so we are just enabling the counter

But just making the enable the counter and depending upon the completion status zero or two will be returned

System dot exit either zero or two zero means successful completion two means unsuccessful completion

So in this way the main function has been written Now it is a high time to create the jar file

So to create the jar file, we'll be going for the respective package and then we shall click

on the export and then we shall give the jar file name, we shall give the path and then next

and finish. So jar file we have already created. So this particular step we are not executing right now

So we shall go for the command in the console. The command is Hadoop

jar so there is a command I've already written so Hadoop jar then we'll be going

for this map produce design pattern jar files the respective jar file path and

the jar file name binning is the there is a package name binning amort

task is a class name input post links will be the this the binning is the package

name and these are class name and then input and post links

this is the path where the the XML file is residing and the output path has been given Now let me execute the command Let me show you that it is really getting separated by the link type ID and here we having two link tabs ID that is a one

and three so let me see the output folder at first let me see so going for the

output folder so going back and then output folder yes so here we're having

the duplicate M one file linked M another file is there to two power power power

files is there part file also has got grid created but it is having the zero bytes

so that indicates that nothing has been written in this part file so we'll be

concentrating on this duplicate and linked these two files were concentrating so let me see the content of them I hope that in the link will be having the

link type ID 1 all the records and in because of duplicate will be having the ring

tab link type IBD 3 so here the command I'm executing first one executing the

I wrote this comment earlier so I'm taking that one from the history so you can find

that all the link tab ID is equal to three for duplicate part duplicate output file

all the link tab ID is equal to three you can find this one and for the other one

yes for this one if we execute we can find that If I execute this one you'll be getting the link type ID is equal to one yes it is the current content

So in this way you have shown you that how to create such a binning pattern example step by step into details

So now we can delete the output folder so that the next map D2 stocks can be executed and I hope you have enjoyed the video

Thanks for watching