Spark Wordcount Example

in this video we are discussing spark

what count example in case of what count

we were having one text file and the

text file is containing multiple

different texts and we were supposed to

count that how many times one word has

occurred in the respective text file so

let us go forward some further

discussion on it and you shall also give

you a practical demonstration on this

problem how to implement what come

problem in spark using the Java

MapReduce program we have already seen

that how to count the frequency of words

in one or more than one text files so

for this example we are going to count

towards on the same file which you

selected in our MapReduce program and

which is used in MapReduce example

earlier so the file is stored on to the

HDFS and now at first we should start

about Hadoop before accessing this HDFS

files so let us go for one practical

demonstration to show you that how this

Watkyn problem can be written and can be

executed my hard-up system is on so it

is running so now here you can find that

here you are having the Hadoop root that

is the HDFS root now under this we are

having one folder that is a Hadoop my

files so there is the heart of my files

under this folder we are having one file

that is a sample underscore file dot txt

so let me show you what is the content

of the file so we shall go for control

alter T we shall open one terminal and

now we shall go for HDFS DFS - CAT slash

Hadoop my files we shall go for cat

Hadoop my files and the file name is

sample file dot txt so file image sample

file dot txt so I'm going to see the

content of the file the file content is

this one so this is a content now we

shall open our spark cell and in the

spark shell we shall execute a program

that means not a program but the state

of statements will be writing some lines

will be writing here one by one to

perform the word count problem on this

sample underscore

file dot txt so that is a purpose and

that is a demonstration we are going to

give you right now so let me let me go

for the initiation of the spark shell so

to initialize the spark shell we shall

go forward spark shell if we initialize

the spark shell then the Scala prompt

will be coming so at first we shall

create one file which will read all this

sample underscore file dot txt content

so let me open that file so let the

Scala prompt come the Scala prompt has

come so I shall go for bar sample file

is equal to SC Assistance for spark

context so text file and then we shall

give the path should be enclosed within

double quotes so HDFS then hold on / /

localhost colon 9000 so 9000 is the port

number and the folder is Hadoop my files

how do my files and the file name is

sample sample file dot txt so this is

the total path with the filename okay it

opens the txt file stored in the HDFS so

now to see the content of the text file

as an array so we shall go for sample

sample file dot collect we shall go for

sample file dot collect so to see the

content of the text file as an array so

you can find that the content is getting

shown you see the content is coming in

the form of an array you can find this

one in the form of an array now we shall

we shall split this particular content

so to split all the words which will be

separated by the blank spaces okay so

what you shall go for this impart and

then W count I can give other name also

no issues so then sample file dot flat

map

so m capital not f flat map and then

line so line dot split and here the

delimiter will be a space so I'm

enclosing this space within double

quotes and now closing it okay to see

the contents inside W count so all words

will be separated in the array so we

shall go for W count not collect you see

in the array all the words have got

separated you can find the output here

I've just marked it you can find that

output okay now to see the contents

inside the W count we issued the command

that is a W count dot collect here so

now we shall put one after each word in

the W count or DD so how to do this one

so we shall go for save hard a map

output I'm just writing this one as map

of P Matt map output W count W count dot

map and then W and here each and every

word will have one after it

I'm pressing enter to see the output

what is the value will be getting a key

value pair type of thing so let me show

you that one also so map output dot

collector so it will show show us the

key value pair type of thing you see it

is the key value pair so key is here

this and value is one so for each and

every word we have a test we have

treated that word as a key and the value

is one here okay now what we shall do we

shall call the reduce by key method we

shall call the reduce by P methods so

now let me go for and that one so Val

reduce or output I shall make this one

as a reduce for P then map of P dot

reduce

by key so by P reduce by key and here we

shall go for underscore + underscore now

what is the final output we have called

the reducer also so what is the final

output so initially we have the we had

the map output now we are having this

reduced output so to get this one I

shall go for reduce output dot collect

you can find that it is coming like this

you see it is coming like this so here

each and every key is there when the key

is unique not having the occurrences

again so it is having the frequency one

count is one but when this particular

key has got repeated for multiple times

so so respective counts are coming so in

this way where you are getting this so

now let me play let me in save the

support on to some file in the HDFS so

how to do that one so reduce reduce

output dot save as text file so there is

a method is this one so we are going for

and passing this parameter what is the

parameter that is a path so HDFS say me

make making this one as / / localhost

colon 9000 we wrote this one earlier

also so now let me decide some path some

directories which will be created so let

it be a spark output /wc spark

before going for that let me show you

that there is no folder called spark

output okay so there is no folder called

spark output or under that folder

obviously the WC spark folder will be

created but there is no folder called

spark output so what we can do so now if

I if I execute this one so there is a

reduce opie dot save as text file so we

are giving this total path with a file

name so we are giving the path actually

here the file will get created

automatically and that should be

enclosed within double-quotes so just in

putting enter so now let me show you

that the corresponding spark output has

got created under this we're having this

WC spark and under this were having this

underscore success and part - 5 0 so

this is the file which is containing

actually the output so let me show you

the output also so how to show so I

shall go for the terminals 1 second okay

so this is the terminal I'm having so

I'm going for let me come out from this

so I shall go for exit so coming out so

I've got the dollar prompt back again

clear so let me see so I shall go for

say HDFS DFS - cat to see the content

here so - cat slash so spark output so

there is a first folder under which were

having the next folder is our there is a

WC spark you can find here so WC spark

and then we are having this part - one

two three four five zeros so that is a

content we are going to get so this is

the content you can find that this is a

content so the content has been written

on to this part - five zeros so here the

content has got written so instead of

writing this fool instead of writing

this full also you can put the what

should I say that respective wildcard

characters we can put so that will also

work for us so that will not also

produce the same

so in this demonstration we have given

you the idea that what are the different

steps should be followed to execute in

the word count problem on to a text file

in our spark shell thanks for watching

this video