MapReduce and Design Patterns - Structure of Numerical Summarization Pattern

Structure of numerical summarization pattern

We know that whenever we're dealing with a big set of data, then those data are to be

categorized or to be clapped depending upon some criteria, and then we are supposed to

calculate the summary on the numerical values. So summary means some statistical functions

So let us discuss structure of numerical summarization pattern. The numerical summarization pattern has mapper

the combiner and the reducer and this combiner is optional so the mapper finds the keys that

consists of each field to group by and the values consisting of numerical items so here the

mapper will be dealing with some key value pairs as usual and mapper will find the keys

that consists of each field of group by and values will be consisting of the respective

numerical items because until and unless the values are numerical we cannot

apply any statistical functions over them these outputs are intermediate and these will not be stored onto the HDFS the mapper output does not get

stored onto the HDFS because it is the intermediate output so its output will be

saved on the local disk the compiler is optional and it is good practice to

use combiners to decrease the overhead of the reducer. So, combiner will do some reducing task before the reducer. So, that's why

combiner can also be called as mini The combinator usage is optional but if we use the combiner then obviously that will reduce the over rate on the reducer The combiner reduces

the number of key value pairs of each split and sends it to the respective reducer. So

combiner will do some clubbing. So we'll calculate some statistical values out of some set

of similar keys. So that's why it is doing a reducing operation

prior to the dataset is made available to the actual reducer. So, here you can get the respective idea

So, here we are having multiple mappers, and these mappers will produce the key and

the summary field values like in this way. And they will be available to the respective partitioner

And then this partitioner will do the suffling and sorting, and the values will be, the

key value peers will be available to the respective reducer, and reducer will produce the

final summary summary of the data depending upon the same key values. So, here this summary

will be obtained by applying some of the statistical functions. So the numerical summarization

pattern structure we have shown here in this diagram. So finally, the reducer phase

it receives the numerical values associated with the keys to perform the actual reducing

task. And that is the structure of numerical summarization pattern. So, in this way, using this particular diagram, we have got the idea that how this numerical

summarization pattern is actually playing its role in our MapReduce applications. Thanks for watching this video