MapReduce and Design Patterns - Total Order Sorting Pattern Example

Total order shorting pattern example

So in this particular video we shall go for the implementation of this design pattern

So in this example we will take the XML file that is a users.xml file

and to short the records based on their reputations. So depending upon the reputation value, the records will be shorted

The records will be obtained from the XML that is users.xml. It also takes one sampling rate to be shorted

rate to make the random sampling. So it also takes one sampling rate to rate the random sampling

because using the random sampling will be going for further splitting. So let us go for one

practical demonstration for the easy understanding of this concept. We are going to have one

example of total order sorting design pattern which is falling under this data organization

pattern here and we will shot users based on their reputation against the sampling rate

of point one so this is the task here we're having the the input file that is XML file

users dot XML under this slash input slash user under this folder we are having users

dot XML having got size 56.18 so this is a folder I'm just marking here 56.18 mb is the size

of this file it is a long file but I've shown some of the records within the user's tag

we are having multiple rows are there these are the user tags are there we are having

multiple rows are there each and every row has got multiple attributes so let me explore

it with this ID reputation on which will be doing the sorting and then creation date

then display name last access date website URL location me then we'll be going for view upvotes downvotes account ID so these are the set

of attributes are there so we shall we shall short users based on their reputation

we're having only one Java file that is the dot order short MR task so

here it is having the the MR there is a mapper class has been extended a

reputation order mapper under this we're having one int writable object that is the output key which has been instantiated we're

going to override the map method within this map method we're having XML

parsed which will be instantiated using XML to map function this function

we have written there in our code which will take one XML and returns one hash

map object which will instantiate this XML parsed string reputation is equal to XML parse dot get reputation so from the hash map using that tag

reputation we are just extracting the reputation here so here you can find that

reputation is there as one of the fields in the row so this attribute is there

reputation we are just accessing that one if reputation is equal to is equal to

null then we are returning otherwise in reputation is equal to integer

dot value of reputation dot in value we are converting that one to the integer and

and output key dot set in reputation and this key value pair we are writing here in the current context context dot write output key comma value so there is a key value pair we writing here so that is for the temporary purpose enclosing it within proper tricatch

block with having the class there is a reputation reducer which extends

reducer it is a very simple class we have written and overriding the reduced

method overriding the reduced method you can find that we're having and that is

our text value which will be iterating over these values and this value is values is nothing but one iterable

class object so context.write value comma null writable. This is my you can see this is my

actually the main function we are going to discuss one by one the main function we are going to

discuss one by one so we are having this there is a class name then input path then output path

and then the sampling rate which you were supposed to pass 0.1. So three arguments are to be

passed. So arguments dot length must be three. We are defining multiple different paths here

So path input path that is initialized with the argument 0 and then partition file will be

initialized with argument 1 and also a concatenating underscore partitions dot LST. So with this

argument 1 we are concatenating underscore partition dot LST so that we are concatenating with it we're having the output stage here

we're having this arcs one there is a second argument whatever you'll be passing and concatenating it with underscore staging so output stage path has

been initialized we're having the output order that is a argument one so that is

the second argument we're passing so on that path so double sample rate is

equal to double. That is the last argument that is the last argument that is above point one

which will be kept converting to double. We're going to have this system file system dot get new configuration

Delete output order true. We're going to delete the output order. We're going to delete the output stage

We're going to delete the partition file and so on. So they are getting deleted

We're defining one job. the name of the job is short users based on reputation so configured job to prepare for

sampling so creating one job the job name is use short users based on reputation

then we are defining the jar carrots the mapper class and at first the

reduce task is zero so performing only the map only job so making the respective

classes assigned and for reducer you are not assigning anything right at this moment set output key and set

output value so this two the respective in writable class so text class were

also defining the respective classes file input format dot add input path that is a

sample job comma input path so whatever you have defined earlier so there are

being used here so there is a sample job and input path see this is the input

path we defined from arc zero sample job dot set output format class that is a sequence

file output format dot class we also mentioning the respective class that type and then sequence file output format dot set output path there is a sample job comma output stage so everything whatever you have defined earlier

we're just using them now what we're doing we're going for the completion status

if the if the completion status is zero that means the successful completion then we'll be

invoking the reducer otherwise we shall not do so if code is equal to zero so we are defining

the job instance and we are setting the the jar class again setting the the job instance we are going for the name of the job

is short users based on the reputation we're setting the the jar class set jar

by class we're setting the mapper class we're setting the reducer class so all

these things which you do earlier also in other programs we're doing the same

now here we're going for set number of reducer three so three reducers we're

going to create so there is the order job set partitioner class that is a total order partition class we are also

defining the respective classes there so Hughes hadoops total order partition class here so under the had to package total order partitioner dot set

partition file that is the order job dot get configuration and the partition

file so here we had just just respect to partition file whatever you

define with this concatenation of underscore partitions dot So here we're going for this. So order job.set output key class in writable and output value will be text class

Order job dot set input format class is a sequence file input format. Class

so set the input to the previous jobs output and sequence file format.set input path that is the order job and output stage

Everything we defined and initials earlier were using them here. Text output format, that is a set output path, that is the order job comma output order

But also going for the configuration, we're also going for the configuration, that is the order job

dot get configuration. There is a mapread. Dot text output format dot separator using the null, that is a null as a separator

That is an empty string actually. It is an empty string. input sampler is input entry string I'm just showing so input sampler dot write

partition file order job comma new input sampler dot random sampler sampling rate

whatever you should be we shall be passing there and comma hundred so we'll be

passing sampling rate as we decided point one there is a sampling rate we're

passing now depending upon the job completion status zero or two the corresponding

running zero or two so that will decide how how the completion has been done so we

shall delete the partition file and output stage file we shall delete them

so false system dot get new configuration dot delete partition file false and

output stage file true and system dot exit code it is a long code you can find that

so many things we are doing I've just explained line by line long code we had to

type yes this is the main main main function is there and then we having this is a taught order shot mr task so now we shall create the jar file so we shall go for the package right button click we shall go for export we shall go for export

then you shall decide the path we shall go for export we shall define the path and

respective the jar file name then we shall go for next next finish so in this way the jar file

will be created but we created the jar file earlier so you are skipping this step

So the jar file is required for the execution of this. So let me go to the console now terminal

So this is the command you are finding. There is a Hadoop

The first line we're writing here Hadoop. Then we'll be going for JAR and then the respective path up to JAR files is a path and

then data organized pattern or jar is a jar file name. Then we shall be defining the respective, there is a package

and then total order shot MR task is the class name then input file folder that is a

slash input slash user and slash output is the output file folder and point one is the sampling rate

so you shall execute the command down so as I told you that we will shot users based on their

reputation and depending on the sample rate that is point one so

but having the multiple files will be created the part files will be created

so let the command get executed completed we shall go to the output folder

so now the file has the program execution has been completed successfully

so let me go for SDFS and then DFS minus CAT slash output slash where I mean this part star and then press enter so part files

has got created how many parts file has got created I shall be going to the

output folder here you can find that all the records all the records in the

shorted from users dot XML all the records have got shorted based on their

reputation user dot XML is a file so based on the reputation

these records have got shorted so we're getting the right file right output so let me go

for the output folder let me go for the output folder to show you I'm just scrolling up

and down so that you can see the content there is the content of multiple part files

so let me go for this output now output folder I'm going for this output folder

so we have created we have got three part files but the first file is having the zero

bytes and other two files having the 32.95 mb and 23.03mb so these two part files are

having the respective contents these two part files are having the respective contents

so I have shown you that how to execute this program how to write the code

explaining line by line so let me delete the output folder there that is a common

practice but optional so that we can execute other map reduce tasks I hope you

have enjoyed the video and thanks for watching