Combiner in MapReduce

In this video we are discussing Combiner in MapReduce

So Combiner will be working in between the mapper and the reducer

The output of the mapper will be the huge one. And if we make those mapper output directly available to the reducer, that will enhance

and that will increase the network congestion. So as a result of that, we were supposed to put one combiner in between this mapper

and reducer. So this combiner can also be called as mini-reducer. reducer. But the position of this combiner or introduction of the combiner in this map

reduce model will be also optional. So let us go for some more discussions on it. So

here you have shown that two mappers are there and the respective reducers are there

So this output, that is the intermediate output of mapper, will not be made available to the

reducer directly because that will enhance the probability of network congestion. So this combiner will be working in between and which you will

be working as a mini-reducer. So, what is combiner? From the Hadoop mapper, it creates a huge

amount of intermediate data, and in the process of sending these data to the reducer creates

a massive network congestion. And the combiners are used to overcome this very problem

And the combiner is also known as the mini-reducer, and it processes the intermediate data

from the mapper which is coming from the mapper and use of the combiner is optional in this map reduce

so how does combiner work so let us go for one example and in this example we can see there is

two mappers and the main data is divided into two parts and the mapper finds the intermediate key value pairs so let us show you the diagram at first So here we having this data and the data has got divided to two different mappers

So, this first line is Hadoop is, Hadoop is, is available to the mapper, and mapper has made a key value pair

where the key is the each and every word, and the value will be one there

So, that is a key value pair. So Hadoop, comma, 1, is comma 1, Hadoop comma 1, haddub comma 1, and is comma 1

Similarly, here in this case, the Mapper 2 will be having this particular data, so sky is cloudy is cloudy

So in this way, it is also producing the respective key value pairs

And these things are nothing but intermediate data. And now this data will be available to the combiner

So this data is not getting available to the reducer directly, but through combiners

So, Combiner is also doing some reducing work, prior reducing work here

So, as a result of that, it is having had doop. So, you see, same key values are there

So the count has got added, so one and one, so it has become two, and ease the same key values

So these two particular key value pairs will merge, and here will be having the value two here

And similarly, the same operations are getting done in Combiner 2. The outputs of this Combiner 1 and Combiner 2 will be made available to the reducer, a reducer

will produce the final output which will be stored on the HDFS that is Hadoop distributed

file system so the output of the mapper is partially reduced by the combiner and in the

last stage all the output of the combiners is again reduced by the reducer and finds the final

output and that will be stored and saved on the HDFS so in this way in this discussion we have

told you we have discussed that how this combiner works in App Produce. Thanks for watching this video