Data Flow in MapReduce Framework

in this video we are discussing data

flow in MapReduce framework in the

previous videos we have discussed how

the mapper is working and next we have

discussed how the reducer is working so

please watch all those videos for the

better understanding of this concept so

here you can find that that is one

diagram is the earth so input will be

there input will be consisting of

multiple different records and those

records will be splitted by the input

split into multiple logical model which

can be handled by the mapper and then

after this split will be having the

record reader which will convert this

records into key value pairs because

mapper can only read and handle key

value pairs then the output of that will

be made available to the mapper and

mapper will be handling this key value

pair the developer can put the logic in

the mapper according to the business

logic requirement or business logic

implementation and then this mapper will

produce a intermediate output and these

intermediate output will not be stored

on to the HDFS but it will be stored on

the local HDD and then then all this all

this outputs from this mappers will be

will be done the suffering and shorting

and that will be stored here and that

will be made available to the respective

reducer so this shorting will be done on

the values stored in the keys and then

the reducer will do the reduce and it

will produce the output that is a part 0

on the HDFS that is Hadoop distributed

file system so data flow in MapReduce

framework the data which will be

processed through the MapReduce are

stored in HDFS and the data are stored

in different blocks and stored in a

distributed format so that means the

data will be stored into multiple

commodity hardware's so steps of data

flow in each mapper one split will be

will be there to process at a time and

developers can put their own business

logic in these mappers and mappers run

in parallel fashion in all of the

missions so mapper will be running in

parallel

between the processing on all of the

missions in parallel and the business

logic will be put in a mapper by the

developer the output of the mapper is

stored on the local disk but not on to

the HDFS and this report is suffle tool

to reduce her to part from the reduce

task when all of the mappers have

completed their task the output is then

shorted and then they will get modest

the reducer takes those data and

performs the reducing task and then the

output is stored onto the HDFS in the

form of part 0 so that is our mapper and

reducer and how the data flow is taking

place we have discussed that one into

details thanks for watching this video