MapReduce and Design Patterns - Hadoop Word Count Example

In this video we are discussing Hadoop word count example. So this is a very fundamental and basic example in our Hadoop environment

Now what is word count example? The word count example. So what count task in Hadoop is used to count the number of words in one or more than one input data set files

So now what will happen at the data set files will be having so many different words will be there

either one data set file or multiple data set files. And from there, this particular problem will take each and every word and ultimately

it will produce the word, comma, the number of times the word has occurred in the same

or multiple data files. So that is known as a word count example

So let there be a file which is containing a sentence that is a duck is a bird

So this is a content in one of the files. So, after performing word count task, it will generate key value pairs like this

So, that is A has occurred for two times. You can find that a duck is a bird

So, A has occurred for two times. Duck has occurred for one times

Ease has occurred for one times and bird has occurred for a single time So this A duck is bird They are the keys and this 2 they are the values So this outcome will be in the form of key value pairs And that is the basic concept

of what count example in our Hadoo. There are two different parts in this task. The mapper task

and the reducer task. After performing the mapper task, it generally, as key value pairs. So, the outcome of the mapper task will be something like this

That means, it will go for each and every word, and then it will produce the value against

that key, the word will be the key, and the value will be one. So, mapper will produce a output

that is a key value pair, but for each and every word, it will not go for whether the word

has repeated or not. It will produce the word as a key, and then comma one as the value

So, in the reducer phase, the reducer takes the same key and count number of values under

the same key. So, whenever we are having the same key, then the reducer will just go on adding the

respective counts there. So, finally, it generates the exact key value pair, and from that pair, we can get

the number of each distinct word existing in the given input. So that is our what count example in Hadoop

Thanks for watching. this video