MapReduce and Design Patterns - What is MapReduce ?

In this video we are going to discuss what is MapReduce

MapReduce is one of the main components in Hadoop ecosystem. In case of MapReduce we are having the mapper methods and the reducer methods

We are having a set of data, there is a huge set of data

And this set of data will be splited into smaller tasks and those tasks will be assigned

to the multiple mappers, multiple working nodes to work on them in parallel

and then the output of that mapper will be obtained as an input to the reducer and now the reducer

will work on them depending upon the customization depending upon the business logic or custom

function the reducer will decide what operation it is going to do on them so it might be some

aggregation operation there and then reducer will produce the output and this mapper is taking

the data set in the form of key value pairs and reducer is also producing the output

in the form of key value pairs. Here the keys are nothing but referencing some files

that is the data set files, and values are nothing but the data sets. So, that is the basic

concept of this map reduce. So, let us go for more discussion with some diagrams on this

topic, that is what is map produce. So here we are having this input, and here we are having

these map tasks, and then we are having the reduced tasks will be there, where the reduced

will be working here it was working as map methods here it will be working the reduce methods

and the final output will be obtained in the aggregated form that is the basic theme behind this

map produce so the map produce is one of the main components of the hadup ecosystem in our

hardup ecosystem video we have discussed there are so many components are there under this

hardup ecosystem there also we had that map reduce ecosystem component map produce is designed to

produce a large amount of data in parallel by dividing the work into some smaller pieces and

independent tasks So large amount of data will not be processed at a time It will be divided into smaller pieces and those pieces will be assigned to the working nodes and those tasks will be executed in parallel for the faster processing

The whole job is taken from the user and divided into smaller tasks and assign them to the working nodes

Map produce programs take inputs as a list and convert to the output also as a list

So it will take the input as a list and it also converts the output in the output in the list. the form of a list. So, let us go for some further criteria. The map or the mapper tasks a set

of keys and values, we can say that it is as a key value pair as input. So, this particular

data will be in the form of key value pairs. Now, questions might be coming in mind that

what is key and what is the value. So, key is actually nothing but a reference to our data

set and values are nothing but the datasets. So, key can be treated as a reference to a data set or reference to a file and the value

is nothing but a dataset. The data may be in a structured or unstructured form and the framework can make it into keys

and values. So, the data set may be in the structured, that means in the form of, say, database and

database tables where the data can be divided or can be represented in the form of rows

and columns and unstructured mid where we'll be going for text files

pre-d-fs we are having the images we're having the videos and they'll be known as

the unstructured data the framework can make it into keys and values the key

are the reference of input files and values are the data sets the user can create

a custom business logic based on their need for the data processing so what

kind of processing will be done that can be customized depending upon the

business need the respective operations will be carried out the respective processing will be carried out on the dataset the

task is applied on every input value now we are going for the reduced task the

reducer takes the key value pair which it which is created by the mapper as input so mapper is taking the input and mapper output will be the input to the reducer and reducer will produce the respective output

accordingly the key value pairs are shorted by the key elements in case of reducer and in the

reducer we perform the sorting aggregation or summarization type of jobs that means here

we are going for some aggregation type of job we can go for a submission we can go for

say counting we can go for say maximum minimum calculations and so on how map reduce task works

so now let us go for the the macro view of the system the given inputs are processed by the

user defined methods all different business logics are working on the mapper section so business

logic will be working at the mapper section and the mapper generates intermediate data and

and reducers takes them as input. As I told you earlier, that the output of the mapper

will be the input to the respective reducer. The data are processed by the user-defense function

in the reducer section. So, the data processing will be done at the reducer section, depending upon the business logic

depending upon the user-defence function or operation, and the final output is stored in HDFS

that is, Hadoop distributed file system, where the final result will be stored there

Now, let us go through one proper diagram here. So, you are having here, HDFS split

So, multiple splits are there. So, Hadoop distributed file system split will be there

So, input will be key value pairs in this way. It will be, the input will be obtained to the respective mappers

So now here the mapping is taking place and it will be, now it will be dealing with multiple

keys and the respective values. So Q1, value 1 to key K to value K

In this way, the mapper will be working. Now their outputs will be coming to this shuffle and short

So here the aggregate values by the key. So depending upon the same key, so they will get aggregated

Then the result will be obtained. That means the output of this shuffle and shot operation will be obtained to the reducer

So these are the reduced methods are working here So we having the Q1 intermittent values and then key K intermediate values will be coming to the respective reducers and then final Q1 and final value and final key K and

final value will be obtained in this way. So, initially we are having this key K, key 1

key K value K, in this way we are having. Then here we are having this shuffle and shot

then the reduced method will be working on it. Then we are finally getting final key 1 value

final key k then final value in this way the things will be obtained as output the

pictorial representation on how the map reduce task works we have shown that one in this

diagram so let us go for another elaborate tip example for the better understanding

so here we're having one example here so dear beer river car car river dear car

beer so these are the value sets we're having so set of values we're having so now

they will be splited in this way. The splitting has been done

Now the mapper is there. So, here the mapping is taking place. So, I am finding deer for one count, beer for count one and river for count one, car for count

one, car for count one, car for count one, in this way the mapping is taking place

Here we are having this shuffling and sorting. So here you can find we are having this beer, only the beer keys are there, only the car keys

are there, only the deer keys are there, only the river keys are there

Then here we are going for the reducing. So this beer has occurred for two

So here the aggregation method is count actually. So here card has occurred for three times, deer has occurred for two times and river has

occurred for two times. So here this is my reducer which is doing the reducing and here we are having this final

result that is beer two, car three, deer two and river two

So here you can find that how the overall map produce what count process is getting executed

into multiple different phases. So this example is a very interesting one and also it is clear our doubts and it is clear

our conception also. The pictorial representation on how the MapReduce task works, we have shown that one

in this way. So in this video we have got the idea that is what is MapReduce

Thanks for watching this video