MapReduce and Design Patterns - Replicated Join Pattern Example

In this video we are discussing replicated join pattern example

So we shall go for the implementation of this pattern. So this program takes posts.xml and users.xml file and joins the records

And we have to select the join type while executing the task

So what kind of joining we're going to do that has to be selected

The join types are the left outer join, inner join and etc

let us go for one practical demonstration for the easy understanding of this concept

in this example we'll be going to implement one replicated joint example which is falling under the

joint pattern here we're having two XML files one is the posts dot XML which will be under the

folder input post and then users dot XML which will be under the folder input slash user

you can find here the post and user folders subfolders are there under the input folder in the

name node so let me go for the let me show you that if you go for the post folder

that is slash input slash post you can find posts dot XML is there now we'll be going

for this slash input slash user and we shall be getting this users dot XML there let me

show you the file contents so we're going for the file content so at first we're discussing

that is a posts dot XML within the post tag we're having the row tags multiple rows are

with multiple IDs multiple attributes ID post type ID these are the different

other view count then body then we're having this say this last edit date so

so many attributes are there so here I just marking that one highlighting that

one for your understanding we're having the tags there is answer count comment

count favorite count community own date and so on so these are the multiple

attributes we're having under users dot XML under the row we are having multiple IDs and all these rows are within this users tag

Users dot XML is having so many different rows But I have shown some of them so abhorts downboards account ID and so on

So in this way you are getting the content of this so let me go for the eclipse we're having only one class here that is a duplicate job

a join MR task it is a map only so no reducer will be there within this class we have defined

one inner class that is a replicate join comment user mapper which extends mapper within

this mapper we have defined one variable one object that is a user map of hash map type

which will take string and string as key value we're having this output value of the type of

text and join type of the type of string so there is a joint type we're overriding the set-up method

In the setup method we are having, we are checking the context

Dot get cache files if it is not equal to null and context. Dot get cache files dot length is greater than zero

Then we just defining one file object that is a user file So this rest of the code will be there User file is equal to new file Dot slash users and then buffer deeded object has been defined So after defining after

initializing that file object that is a user file we are going for this buffer reader object

also. Then within the while before going into the while we're having the line if the line

is equal to buffer deeter dot read lines will be reading line by line XML parts is there

So one map object XML for parts which will be initial as with this XML to map method. So let me go for the XML to map method. This method will take

one XML file as input and returns one hash map object as output. So here we are having this XML

to map method. This method will take XML as input and hash map object as output that will

instantiate our XML parsed. So from the XML parsed, we are going for .get ID. So the ID

will be taken in this user ID. If user ID, is equal to is equal to now then continue I shall I shall not go for the rest part of the code

otherwise user map dot put user ID comma line so we're just putting this this one

user ID and the line content will be put onto this user map we have kept it in the

try catch block we're kept it in the try catch block and you know we're having this buffer

reader we have just made that one close and here you can see that we have joint type

has been initialized with is equal to context dot get configuration. Dot get join. Type. So after closing this buffer reader, we are just going for

this join type. So joint type is equal to context. Dot get configuration dot get join

So regarding this joint dot type we'll be discussing that one later. So here we'll be selecting

only two types of join. One is the inner and the one is the left outer. Now this is my map method

Within this map method we are having this XML parsed. There is a map object. Again it is getting instantiated using this XML

to map which you did earlier also is XML to map I used that one earlier also yes

at the line number 39 the XML to map we're having this string owner ID so

from the XML to XML parts dot get user owner user ID we're initializing owner ID

if owner ID is null then I shall return map will not get executed as part so user

info is equal to user map dot get owner ID so now user

info will be initialized against this particular owner ID the user info if user info

is not equal to null the output value will be set with the user info and we'll be

writing onto the context that is key value paired value and output value so that will be

written onto the context temporarily else if joint dot type equals ignore case left outer if it

is left outer then we'll be writing this one as value comma new text null so if we are getting

the match then we'll be writing the value and output value otherwise we'll be writing

value and null because there is a left outer joint so now let me discuss our main

function let me discuss our main function here we have we require four

arguments so first one is the class name then post input folder user file location output folder then output folder then we having the fourth argument it should be either inner or left outer so if the argument length is not equal to

four then we shall exit now from the fourth argument that is the arcs three will be initializing

the joint type so joint type should be either equal to inner or left outer if it is not that

then error message will be printed joint type not set to inner or left outer or left outer and system dot exit 2 so here the program will get terminated now we shall

define one job instance we shall define one job instance so here the name of

the job is replicated join to posts and users so this is the job name we have

defining one job instance get configuration join type join type join type join dot type comma join file class has been set

jar file class has been set replicated join mr tux dot class so this joint type you have taken this one from the fourth argument as I

discussed earlier also again I'm repeating it should be either inner or left

outer set mapper class will be initialized but before that just see join

type we used earlier here we used the joint or type see the line number in

the line number 54 you have used this joint type in the line number 54

here we have used this joint type now that has been you that has been initialized

by the input argument that is argument number four now we're just setting the

mapper class the mapper class name has been set and reducer there will be no

reducer to reducer count is zero input path has been set with this arc zero that

is the first argument and output path has been set with the third argument that

is the arcs two that is output path so output key class

will be text type output value class will be also text text type text dot class and add job

cache file will be that that one that is a new uri arg 1 plus slash user so it is a we are doing this

concatenation that is a second argument plus this has users and depending upon the job completion

status 0 or 3 the value will be returned so now let me let me create that expected jar file now

So you shall be going for the package and then right button click and then export and you shall give the path and the jar file name as we did for other

cases so then we'll be creating the jar file I have created the jar file already so I'm skipping this particular step

so we are supposed to go for this export there. So now going for the terminal. So here is the command we're having

here is the command. So Hadoop jar then we'll be going for the

jar file name along with the path at first and then join pattern dot jar is a jar file name

Replicate join is the package name So next one will be the replicate join will be the package name and then replicate join a mark task will be the class name

Then we going to have that folder that is the input post and then input user Then users dot XML we are also mentioning users dot XML output folder and here we require the joining as inner

here we require the joining as inner let me execute the command instead of inner

also we can go for left outer because two options were there in our hand so let me go

for the execution of this command you know that in case of left outer join so all

all the all the records all the rows will be coming from these posts dot XML and

those records will be coming from the users or XML whenever there is a match

otherwise that will be printed as null null string is there so all the

records will be coming from the left-hand side there is a there is this first one

whatever you are mentioning and next one is my users users or XML so inner join

means all the records will be coming from the left and those records will be

coming from the right which are having a match so let me go for the content printing

of this part file so that you can see sdf s dfs minus CAD slash output slash part

star yes now let me place enter so this is a content there is a content you can

see the long content are there so all records will be coming from posts those

records will be coming from users for the joining wherever ID will match there is a

left outer join so I've discussed each and every step into details each and

every Java code we have explained I think now you're comfortable to work in

this example let me show you some rows with how the things are coming

Let me mark some of the rows there. Let us consider, but having so many, so many rows are there

All the rows are coming from posts. So let us go for one row marking

Say this one row, there's the last one I'm just marking. ID is 256058 and the other details are there

That is another row which is having a post involved. So that's why it is a long row, the content

In this week and you can go through the content whatever has been produced you can see all the IDs are there whatever there the score the view count the body and so on everything is there

that is that is a complete row comment count and so on this is another row reputation creation date display name last access date views up votes down votes account ID

everything is there so I think now you are comfortable to work in this example

so what you shall do we shall delete the output folder so that we can execute the

map reduce stars so we'll be going for this and then minus rm

minus r and then output thanks for watching