MapReduce and Design Patterns - Inverted Index MR Example

In this video we are going to discuss inverted index map reduce example

So this particular video will be assisted with some practical demonstration to show you how to write

the codes, how to execute it to get the outputs. Inverted index example

So we'll be considering this example here. From the posts.xm. Find the Wikipedia links make the inverted index

So we are having one XML file that is a post.xm. and on which we'll be applying this inverted index

And in this program also find the type of the post using post type ID

And the type of the posts may be like comments, post questions, etc

So there are different type of posts will be there. So let us go for one practical demonstration for the easy understanding of the implementation

In this program, we are going to demonstrate and implement the summarization design pattern

under which we are going to implement inverted index MR task we are having one

XML file that is a post dot XML which is under the folder that is our

slash input slash post so under this particular path we are having posts dot XML

this post dot XML is a huge file you can easily find that it is a huge file it is

having a size of 108.93 mb so we are going to show you that in input slash post

the file is existing and it is a huge one but I shall be showing a sum of the rows there

in this particular file for your better understanding and there is 108.93mb so let me go for

the file content we're opening the text editor so we can find that it is a XML file it is a huge

long file but I have shown some of the records here under the post tag we're having row

tags are there and you can find this is a complete row and it is having multiple attributes

we're having under the row tag we're having the attribute like ID post type ID

accepted answer ID creation date the next one we're going to have the score then

view count then body having the next one is a owner user ID last edited user

ID last edit date last activity date title title then tags then answer count comment count favorite count community owner date so

these are the different attributes are present so what we shall do we shall search

for each and every row we are supposed to find out those respective posts which are

related with the Wikipedia which are related with the Wikipedia so that we are

supposed to find out but having only one Java file here that is the inverse index MRTarx Java So we are having one Java file and this Java file is extending the this Java file is

there within that we are having inner classes to inner classes one will be extending

the mapper class and another one will be extending the the reducer class so this

a mapper class extended that is the invert index mapper having the member ID

that is a member variable ID type and keyword both of them of text type

here we're overriding the map method we are having XML parsed there is a one

hash map object and here XML to map one method is there let me show you the

method at first so this is a method which will take one XML as input and

returns the hash map as output so this is our XML to map so this method will

take xml as input and hash map as output so under this we are having

the tri-block and here we're having this ID we're having main text then body will

you having the body and then we'll be having the next one is the body one and next

one is our dog type so these are the different variables we have defined now let me

discuss that if if the respective body if it is not equal to null that means if the

body is having some content then we are checking the post type if the post type if the

type is not equal to null and if the post type is equals 1 you know that we are

having the post ID also we are going to get and then regarding the post type if

the post type is not equal to null if the post type dot equals 1 then doc type is

question if post type is not equal to null doc type equals to 2 then it is answered

if post type is not equal to null otherwise the doc type will be unknown and if the

body is null in that case with the body is null but if the main text is not equal to null then you shall go for the doc type as comment

otherwise it will return so here we are finding the doc type using this particular logic

finding the doc type then id.contact we're having the where doc type is getting added with this

id but define one yes the doc type is getting added with the idea then we're going to define a

text so if the doc type dot equals comment text is equal to main

text or two lowercase otherwise text is equal to body dot to lowercase so we are

converting them to the lowercase we're defining one string array there is a

text word and from the text using the blank space as the delimiter we are going to

split them and put them in this text array that is the text words now we are

searching for each and every word if the word contains Wikipedia dot word g

then we going for replacing hd f is equal to ampersion code semicolon will be replaced by blank space and amperson code semicolon will be replaced by the blank space again So word dot replace all and word dot replace all ID type

ID keyword dot set word and we're going to write this key value paired on the context

So cTS dot write keyword comma ID type. So this keyword comma ID type is going to be written onto the respective context

We have kept this one in the tri-catch box. lock accordingly now we shall discuss the reducer we shall discuss the

deducer the class name is in a invert index reducer we are going to

override the method reduce we have passed the parameters you can you can see

that one we are defining one string builder class object and using this

lithium we are just going on adding or appending with this string builder class

object that is our STR built there is a row ID dot to string plus a blank

then output value dot set which we which is which we define as a text type we are

going to set that one and then we are writing this key value pair onto the

context so we're writing this one we are just taking the substring starting from

the index 0 to stuard bill dot length minus 1 converting it to the string we're

writing that one onto the context as key value pair this is our main function this

main function is taking the common line arguments and it is checking whether the

length is two or not if the length is not true then one error message will be

printed and system will exit then you are defining one job object we're defining

one job object giving the job name and so on and we know that respective class

is there there is a set juror by class we are going for input file for file

input format and file output format we know that argument zero will be containing the

input file path and one will be containing the output file path where have been

this set mapper and set reducer and here we'll we define the respective inner classes so there is a set mapper and set

reducers so these two classes will be assigned to the will be assigned to the

job and then job dot set output key job dot set output value and then we're

having the success if the success we are getting then zero will be returned

otherwise one will be returned so same thing whatever you did in the earlier

programs also you can pause the video you can watch it now what we shall do we

shall go for creating the jar file there so what we should

shall do creating the jar file so in this case we are going to now we shall go for export

we shall go for export then you shall go for export and you shall go for this jar file

then you shall go for the respective path and the file name has to be given properly

then next and finish but already we have created the jar files so we're not going to do that one

but these are the steps to be to be followed to create the jar files and

And then we shall come to the command prompt and this is the respective command to execute the code So Hadoop JAR so this is the respective path and then the JAR file name then inverted index

is the package name, then inverted index MR task dot class is the class name

Input slash post is the input file path and output is the output file path

So we know that output file will be having the name as part. r.0s will be the output file name. So if I execute this code, this command, I think it

would works successfully but no, it is, it didn't work. So I feel that that name node is in

the step mode. So let me take out from the shape, shape mode now. So let me issue the command

and that is your Hadoop and then DFS. admin then go for shape mode then leave pressing enter you can find that it has

come out from the set mode so let me execute the command once again I think I

hope now the output file will be created under the output folder with the

name starting with part yes it has got created so let me show you the respective file system you can find that we have gone for

and the input post my present directory so I've gone for the root you can find that

under the output folder root output folder we're having part hyphen r hyphen zero so

this is a file which is containing the respective output so let me execute the cat

command that is the HDFS dFS minus cat slash we shall go for

for this output we should go for post part star yes so all the files starting with the part

will be printed you can find that here we're having this the outputs so here we're

getting all those links where Wikipedia is present so 1 24 314 plus answer you can

find that the first part there is a 988 06 is the ID and plus question is a doc type

which you decided using that nested if statements So we can find that we are having this is the respective post link and each one of them is containing the Wikipedia and this is the ID plus the doc type

So this is the respective output or whatever you have got. So let me delete the output folder here

So this is output you can you can easily see. So HDFS DFS minus RM minus R then output folder we are going to delete

So folder has been deleted. Thanks for watching