What is MapReduce? Hadoop

In this video, we are going to discuss what is MapReduce

MapReduce is one of the very important and major components in Hadoop ecosystem

Whenever we are having a large set of data, a huge data set, then in that case the huge

data set will be divided into smaller pieces and processing will be done on them in parallel

in MapReduce. And multiple number of working nodes will be there on which these small pieces of this

datasets will be assigned and the processing will be done in parallel and then the result will be

obtained in the form of a list. MapReduce takes list as input and produces list as output

and that is the main purpose of MapReduce. So, let us go for some more discussion on this

map reduce. So, here we are having this input and here we are having these map tasks and then

we are having the reduced tasks will be there where the reduced method will be working

here it was working as map methods here it will be working the reduced methods and the

final output will be obtained in the aggregated form that is the basic theme behind this map reduce so the map MapReduce is one of the main components of the Hadoop ecosystem In our Hadoop ecosystem video we have discussed

there are so many components are there under this Hadoop ecosystem. There also we had that

map produce ecosystem component. MapReduce is designed to produce a large amount of data in

parallel by dividing the work into some smaller pieces and independent tasks. So, that you

large amount of data will not be processed at a time it will be divided into smaller pieces

and those pieces will be assigned to the working nodes and those tasks will be executed

in parallel for the faster processing the whole job is taken from the user and

divide it into smaller tasks and assign them to the working nodes map produce programs take

inputs as a list and convert to the output also as a list so it will take

the input as a list and it also converts output in the form of a list

So, let us go for some further criterias. The map or the mapper tasks a set of keys and values we can say that it is as a key value pair as input So this particular data will be in the form of key value pairs Now questions might be coming in mind that what is key and what is the value

So, key is actually nothing but a reference to our data set and values are nothing but

the data sets. So, key can be treated as a reference to a data set or reference to a file

and the value is nothing but our dataset. the data may be in a structured or unstructured form and the framework can make it into keys

and values so the data set may be in the structured that means in the form of say database

and database tables where the data can be divided or can be represented in the form of rows

and columns and unstructured mid where we'll be going for text files predfs we are having the

images we're having the videos and they'll be known as the unstructured data the framework

can make it into keys and values the key are the reference of input files and values are the

datasets the user can create a custom business logic based on the need for the data processing

so what kind of processing will be done that can be customized depending upon the business need the respective operations will be carried out the respective processing will be carried out on the dataset The task is applied on every input value

Now we are going for the reduce task. The reducer takes the key value pair which is created by the mapper as input

So mapper is taking the input and mapper's output will be the input to the reducer and reducer

will produce the respective output accordingly. The key value pairs are shorted by the key elements in case of reducer

And in the reducer, we perform the shorting, aggregation or summarization type of jobs

That means here we are going for some aggregation type of job. We can go for, say, submission, we can go for, say, counting

We can go for, say, maximum minimum calculations and so on. How MapReduce task works

So, now let us go for the macro view of the system. The given inputs are processed by the user defined methods

All different business logics are working on the mapper section. So, business logic will be working at the mapper section, and the mapper generates intermediate data and reduces takes them as input

As I told you earlier, that the output