MapReduce and Design Patterns - Cartesian Product Pattern Example

In this video we are discussing Cartesian product pattern example

So here we shall go for one implementation of this concept. In the previous video we have discussed what is the Cartesian product design pattern

So now let us go for the implementation through one example. So what is the assignment

Assignment is something like this. So in this example we have designed to find the cross product of same data file and we are

providing the tax dot XML file to find the cross product so here we'll be using

only one XML that is a tax dot XML and we can use any other file but as this

process is costly so we are providing the XML which is the smallest

in size so now let us go for one practical implementation of this concept to

get the idea in a better way under the category of joint pattern design pattern

we are going to discuss the implementation of Carditionvox product example and we shall discuss

we shall handle this tags.xml. So that will be our input XML file under the folder

slash input slash tags. So this is one XML file will be going and we shall go for the

Cardition product operation. So let me show you the current content of this tags.x.m. What are the

tags are there and residing under this tags inputs tags so let me go for this

tags dot XML so under the tags we're having multiple rows and here I'm

highlighting the respective attribute under the row tags so ID tag name count

expert post ID and then we keep post ID so these are the respective

multiple rows are there in this tags dot XML let me jump to our Java file it is

having only one Java file that is a Cardition product MR task so Cartesian product

Amr task we're having multiple inner classes under this class the first inner class is

Cardition input format which extends the file input format so here we're defining

some public static final string variables and we're having the variables like your

left input format is equal to cart. left dot input format and all this all this

right-hand side whatever I were writing there available under the Hadoop API so left input format card

dot left dot input format then we shall be having our left input path so another

public static final string so left input path that will be initialized with

card. leftته right input format right input path so all this public static final

strings will be initialized here so we have been initialized them all the four

then here we're going to have one that is a set left input one method we're

defining and here we're writing only two that is a job point or set left input

format which will be set with input format dot get canonical name a left input path

will be set with input path so that is the two lines we have written under this

set left input these two lines we have written I'm just marking so that you can understand

And now we are having this set right input another method we have written under this class

so here we are having two lines job convent set right input format input format

Input format dot get canonical name and right input path will be initialized with this input path

Next we are going to overwrite one method that is the get splits

We're having the method get splits where overriding that one it returns in input split

array it returned the input split array. So here we're defining this input split two objects, left splits and right splits

So here you can see that get input splits, confdot gate, confdot get left input format and

the left input path, right input format, right input path, and numb splits and numb splits

So that's we have passed as the input argument, we have just mentioned that one there

Next one we're going to have the size of final split, that is the left size into right

sides that is obvious because in case of partition product we are going to have

all the possible combinations of left schema and right schema so the length

will be multiplied as we have mentioned here there is a left splits dot length

into right splits dot length and number of columns that is a number of

attributes will be added in the in the final result but the number of rows will

be multiplied in the final result that is a that is a feature of this cardition product so here we're executing I nt I is equal to zero input split

left item will be searching on this left splits so input split right item will

be searching on the right splits so for I is equal to zero we are executing this

to nested for loops so final split I is equal to new composite input split to

so we're having we're going to have this left item and right item we're going to

add here using this lines and then i plus plus or plus plus in this way we shall go on

adding the final split will be populated with this left item and with this right

item and final split i will be at first instantiated with this new composite

input split to so in this way the for loop has got executed and marking so that

you can understand it better i plus plus will be done for each and every

for loop execution and we are writing one log in only to for the final check that final splits to process is this so that will

be printed when you shall execute our code and return final split everything has been kept in the tricatch block for the exception handling this final

split dot length will be printed when we shall show you the output there and the

value will be returned now we are going for another method that is a get record reader get record reader which will return the Cardition record reader object which return the

Cardition Record Reader object and here it will be having the composite input

split converting the split there is a typecasting we're doing and the job

conf and the reporter so these three parameters will be returned against this

particular object we have created under the class Cardition Record Reader so I'm

just marking that one you can see that from where the values came so they came from the parameters input parameters now the next method is get input

split so in case of get input split we're defining one file input format object

that is the input format and this particular has got instantiated this

object has got instantiated using reflection unit needs dot new instance class dot for name there is the input format class comma

job conff so against this particular class the new instance has been created and

the job conf which is passed as the input parameter so they have been mentioned there input format dot set input paths there is a job configuration

there is a job conff and input path so input path is there that that has been

passed as input string when you'll be calling these functions you can see the

parameters will be passed and then we're returning the input format got gets

plates job conf and number of splits so in this way the methods are ready we're

having another inner class that is a partition record reader which implements the interface record reader and under this record reader we're

having one method the methods are to be instantiated but defining one

private record reader object initialized with now and left record reader and the right record reader initialized with null store

configuration to recreate the right record reader so file input format under that

we have created one object that is a right right I input format then we are

having the job conf object there is a right conf input split object there is a right

split is so reporter object that is the right reporter so these are the store

configuration to recreate the right record reader we're going for the helper

variables like a left key left value right key right value and we are

initializing the next left with true and done with false so these are the

bullion variables are there they are known as a helper variable so all all this

we have defined under this inner class actually this is one this is one you can find that there is a condition record reader is one

inner class so this is a constructor under this class this is a constructor so

this constructed has got multiple arguments and those arguments are being passed to initialize this right conf right IS and write reporter so this

these these variables are getting updated by the input parameters passed to this

right record reader constructor you can find that I'm just marking so that you can

feel that from where the values are coming and next one is the right reporter

which will be initialized with this reporter which we passed as the input argument to this constructor now we have kept the block within the try-catch

file input format left file input format so left left FIF we're

initializing that one passing the that is a reflection you tools dot new instance

So this class.4 name, there is a conch. .get condition input format, dot left input format, and then conf

So in this way we are just initializing some variables and all these variables will be required

for the left record reader. To create the left record reader will be just creating all this

So left record reader is equal to left FIF get record reader split

get 0 that it means you for the we are going to split that one the first

argument we are going for splitting conf and that one there is a reporter these

are the passing arguments now going for the right input format we are

going for the right input format so here we are having this one the class dot for

name is having the con dot get quotation input format right input format and conf

So some same thing which you did in our left FIF here also are doing the same then we are going for this right record record reader has been initialized accordingly

So everything has been kept in the tri-catch block I'm going to define create the key value pairs for parsing

So left key will be initialized with thismaan left value will be initialized accordingly create create and create value then

right key and right value will be also initialized with the same way as I have marked

here now here with overriding few methods that is a public text create key which

returns a new text object that is a create key will be referring there next

one is a create value which will be returning a new text object we're having

this public long get pause throws I you which returns left rector reader dot get pause so that will be returned

there overriding the method next within which we are having this text key and

that takes value so now here we are doing this next method body so if the left next is present that means not null and if and not left record reader then done is equal to true break

That means it will come out from the loop and that will be done for the next key value pair

So if the left record record rector dot next is not existing, then done is equal to true and break

Else key dot set left value dot to string. so it will be converted the value to be converted to the string so key will be set and next

left is equal to done is equal to false so in this way we are just written this code so reset

the right record reader you are going for the next line that is a reset the right record reader

you can find each and every line has been associated with the respective comments right record

reader has been initialized with this right input format dot get record reader and record parameters

have been passed if right record reader dot next is existing these are the

parameters I'm showing that one just see these are the parameters are there

we have passed that one so if right record reader dot next write key and right

value then we're having that if block there so value dot set value dot set there's a

right record reader dot next right key and right value if it is not null then value dot set right value dot two string converting it to the

string and assigning to the value else next left is equal to true so completing

the right go to the left again so while next left is not equal to true so in

this way the things will be done and return not have done that means if done

is equal to true then it will be returning false if done is equal to false then it

will be returning true so that is the bully that is the output will be there

so the function will have a sum output that means return argument now overriding the close method we are not doing anything just

we are closing the respective left record reader and right record reader we are

closing here so these two lines were written under the close over it in method

now we are having this get process so return left record reader dot get

process so that is our get process we have also over it in the respective

method now let me go for the mapper let me go for the mapper class so now here cardition pattern mapper extends the map reduce

base and implements the interface mapper you know that whenever you are going to implement the

mapper interface we're supposed to write you must be writing the map method for that

because the map method is undefined in mapper interface so here this is the case now we'll

be going for private text there is output key is equal to new text we're defining one new text and then the

map function we have written after defining this output key we have written

the map function within the map function we have written that key dot two string

dot equals values value dot two string if it is not true when both are not equal

then you are defining two string arrays one is a left left okay and everyone is

a right talk that is a T okay you can find that so they are getting

initialize with this key to string so left to key here slash slash s is actually

the regular expression that is a blank space actually actually the blank space

in regular expression we express it as slash s but we are writing that one within

double course so we're writing that one slash slash s so left to key and right

to key you can find by defining two hash set objects there is a left set and

right set these are the two hash key has set objects we have defined

and instantiating arrays to as list. Left to Key and Right to Key

So INT same word count is equal to zero. String Builder object were defined as words is equal to new string builder

We're executing one fall loop. That means a string element which will be searching iterating on this left set

If right set contents element, if the element is also available on the right set, add element

from right set so words dot append element plus comma so comma will be the delimited

here and the same word count will be increased by one if the same word count is

greater than two if it is so and after executing this one if same word count is

greater than two then output key dot set words plus one tab and then key will be set

to the key and output dot collect will be the output key and the output key and the

value. So I'm just marking them so that you can understand that output key output

that output is a goal output key comma value. Now let me come to the main function. Within

the main function we have defined one long start time which is the current time

in millisecond the current time millies that is the respective class under that

we have instantiated that is a method we are called and return argument has

been put on this long start time and then it is a long type so job configuration

config is a new job configuration cardition product but define one job with the name

and here we require two arguments to be passed so usage is given their

condition product MR task that is a class name input folder and output

folder the input folder and output folder is to be given if the length count is

not to then error message will be shown showing this usage but defining the set

jar by class set mapper class but defining each one of them after after

discussing this set jar by class there is a jar class name set mapper

class in a class we define already that is a condition pattern mapper dot class and here it is a map only job so no reducer now going to initialize this set input format that is a

cardition input format dot class set left input there is a config text input

format dot class and argument zero that is a first argument and set right input

so that will be also taken from the argument zero that is the first argument that is the input path text output format set output path

config and new path arcs 1 that it means that is a second argument whatever

will be passing that is a there is an output file path that will be there

config dot set output key class and config dot set output value class so all of

them will be text class job will be made running so job client dot run job config and if the

job is not complete that is not is there complement is there then you shall wait

for one second slip thread or slip 1,000 milliseconds finish time will be

calculated finish time so define one a long finish time system dot current time

millies is a respective value will be there so now we'll be just going for the

difference of finish time and a start time to get the total time in millisecond and if the job is successful return will be zero otherwise return will be two

So finish time and start time will be the total time taken in millisecond

So that will be calculated here. So now we shall go for the jar file creation

So you know, package, right button click, package name, right button click, then export

Then jar file will be created giving the jar file name and the path

So that we did so many times in the earlier programs also. So same process will be executed

But we have already created the jar file. So we can skip this path here

Skip this part here. now you can directly go to the console to show you that how to run this program and

how the outputs would be coming so that will be a better idea for us so here the

command we have listed here there is a Hadoop jar then the respective jar file

folder and the jar file name then we'll be having let me come down yes let me

having this one as the and there's a package name next one is a class name then

we are having the input path that is a slash input slash tags there is a path where the xml file

is residing and the output path is there if i execute my command you can find that we shall be

executing this comment now so let's see the time in millisecond is crear 3 5374 we wrote the time

there is a finish time and a start time so 35374 is a time in millisecond

Now we shall show you the output that how the cardition product has been done

So the command is HDFS, then you shall go for minus CAT, then we shall go for blank space

will be there, blank space and then output. And then you shall go for there is a part star

P at this start okay so now if I press enter you know that in case of

condition product between two data sources the number of columns will be added

and a number of rows for two data sources will be multiplied in the result

and table so huge number of records are there you can find that is a big

output the part is containing so big output you can find so big so long huge

outputs are there because the number of rows are getting multiple it is producing all possible feasible combinations of two record sources so

that's why it is a big output we're getting you see for the couple of multiple

number of seconds it is just scrolling up and the outputs are getting printed

so I hope that you got the idea how to write the code how to execute the

commands how to get the outputs how to see the outputs and everything we have

done step by step so I think now you are confident to work on this Cardition product example falling under this joint pattern you

can do it at your end also pause the video type the code and get it done

is a long output we are finding all possible combinations of two record sources

number of rows will be multiplied number of columns will be added up in the final outcome

that is the basic logic behind the cardition product so Cartesian product will

be used as when required not always because it will cause it will occupy huge

memory so that's why you always try to avoid it so this is a huge

huge use our output we have got you can find that I'm just putting that output

on the screen let me see the file size is how it is the file size here so let me go

for the output folder let me go for the output folder the input file was here

one zero six point two six kb the tags dot XML let me go for the output folder

see for the output folder the file size is 296.15 mb so you see it has got so

much big in size compared to the input file size input file was in kb it is now in

mb so let me see the output here let me delete this output folder thanks for watching