What is MapReduce?

in this video we are going to discuss

what is MapReduce MapReduce is one of

the main components in the Hadoop

ecosystem in case of MapReduce what is

happening we're having a huge set of

data or data set that very data set will

be divided into multiple smaller parts

and they'll be assigned to the working

nodes so that the processing can be done

in parallel in a faster way on the

working nodes the mapper methods will be

working and this particular method

mapper method will be working on the

chunk of the data set or the part of the

data set assigned to that this input

will be in the form of key value pairs

key means the file references and values

means that data sets the output of this

mapper methods will be obtained as the

input to the reducer method the reduce

it will execute some functions these

functions are the custom functions

depending upon the business need some

functions will be operating on this

particular output of mapper methods and

those functions will do some aggregation

some processing on data then the reducer

will produce a final output and that

will be also in the form of key value

pairs and that is a basic concept of

this MapReduce so let us go for further

more discussions with some diagrams and

examples to discuss what is MapReduce so

here you are having this input and here

we are having these map tasks and then

we're having the reduced tasks will be

there whether reduced method will be

working here it was working as map

methods here it will be working the

reduced methods and the final output

will be obtained in the aggregated form

that is the basic theme behind this

MapReduce so the MapReduce is one of the

main components of the Hadoop ecosystem

in our Hadoop ecosystem video we have

discussed there are so many components

are there under this Hadoop ecosystem

that also we had that MapReduce

ecosystem component MapReduce is

designed to produce a large amount of

data in parallel by dividing the work

into some smaller pieces and independent

tasks so large amount of data

will not be processed at a time it will

be divided into smaller pieces and those

pieces will be assigned to the working

nodes and those tasks will be executed

in parallel for the faster processing

the whole job is taken from the user and

divided into smaller tasks and assign

them to the working nodes MapReduce

programs take input as a list and

convert to the output also as a list so

it will take the input as a list and it

also converts output in the form of a

list so let us go for some further

criterias the map or the mapper tasks a

set of keys and values we can say that

it is it is as a key value pair as input

so this particular data will be in the

form of key value pairs now questions

might be coming in mind that what is key

and what is the value so key is actually

nothing but our reference to our data

set and values are nothing but the data

sets so key can be treated as a

reference to a data set or reference to

a file and the value is nothing but our

data set the data may be in a structured

or unstructured form and the framework

can make it into keys and values so the

data set may be in the structured that

means in the form of say database and

databases tables where the data can be

divided or can be represented in the

form of rows and columns and

unstructured mid where will be going for

takes files PDFs we're having the images

were having the videos and they'll be

known as the unstructured data the

framework can make it into keys and

values the key are the difference of

input files and values are the data sets

that user can create a custom business

logic based on their need for the data

processing so what kind of processing

will be done that can be customized

depending upon the business need the

respective operations will be carried

out the respective processing will be

carried out on the data set the task is

applied on every input value now we are

going for the

tasks the reducer takes the key value

pair which it which is created by the

mapper as input so mapper is taking the

input and mappers output will be the

input to the reducer and reducer will

produce the respective output

accordingly the key value pairs are

shorted by the key elements in case of

reducers and in the reducer we perform

the shorting aggregation or

summarization type of jobs that means

here we are going for some aggregation

type of job we can go for say summation

we can go for say counting we can go for

say maximum minimum calculations and so

on how MapReduce task works so now let

us go for the the macro view of the

system the given inputs are processed by

the user different methods all different

business logics are working on the

mapper section so business logic will be

working at the mapper section and the

mapper generates intermediate data and

reducers takes them as input as I told

you earlier that the output of the

mapper will be the input to the

respective reducer that data are

processed by the user different function

in the reducer section so the data

processing will be done at the reducer

section depending upon the business

logic depending upon the user different

function or operation and the final

output is stored in HDFS that is Hadoop

distributed file system where the final

result will be stored there now let us

go through one proper diagram here so

you are having here is the FS split so

multiple splits are there so Hadoop

distributed file system split will be

there so input will be key value pairs

in this way it will be the input will be

obtained to the respective mappers so

now here the mapping is taking place and

it will be now it will be dealing with

multiple keys and the respective value

so q1 value one two key k2 value K in

this way the mapper will be working now

their outputs will be coming to this

saffle and short so here the aggregate

values by the key so depending upon the

same key so they will get aggregated

then the result will be obtained that

means the output of this

shuffle and short opera

we'll be obtained to the reducer so

these are the reduce methods are working

here so we are having the q1

intermediate values and then key K

intermediate values will be coming to

that respective reducers and then final

q1 and final value and final key K and

final value will be obtained in this way

so initially we are having this key K q1

value 1

he came value K in this way we're having

then here we are having this suffle and

shot then the reduced method will be

working on it then we are fine finally

getting final key one value final key K

then final value in this way that things

will be obtained as output the pictorial

representation on how the MapReduce task

works we have shown that one in this

diagram so let us go for another

elaborate example for the better

understanding so here we are having one

example here so deer beer River car car

River dear God beer so these are the the

value sets were having so set of values

were having so now they'll be splitted

in this way the splitting has been done

now the mapper is there so here the

mapping is taking place so I'm finding

dear for one count beer for counter one

and River for count one card for count

to one car for count one and river for

counter one in this way the mapping is

taking place here you are having this

suppling and shorting so here you can

find we're having this beer only the

beer keys are there or the car keys are

there only the deer keys are there only

the river keys are there then here we

are going for the reducing so this beer

has occurred for two so here the

aggregation method is count actually so

here car has offered for three times

deer has occurred for two times and even

has occurred for two times so here this

is my reducer which is doing the

reducing and here you are having this

final result that is beer two car three

deer two and liver two so here you can

find that how the overall map produced

what count process process is getting

executed into multiple different phases

so this example is a very interesting

one and also it is clear our doubts and

it is cleared of a conception also the

pictorial representation on

how the map produced a squawks we have

shown that one in this way so in this

video we have got the idea that is what

is nap reduce thanks for watching this

video