Spark Commands

In this video we are discussing Spark commands

So this video will be assisted with one practical demonstration also for your understanding

So let us discuss some of the Spark commands which we use on very regular basis

So as the Spark is developed in Scala, so we have to use Spark commands in Spark shell and

this shell comes with the Scala prompt. So to start the Spark, we have to use the Spark

spark hyphen shell command so how to use this one how to invoke this one so at the dollar

prompt you are writing this spark location the respective path and then slash bin slash spark

hyphen shell then the scalar prompt will be obtained now to create a dd from a file so how to

create one oddddd from the file so well text file is equal to a c that is a spark on text

and here we are having dot text file and the file path so

using this command we can create one IDD one RD with a file and here the name of the file

with the path should be given within double quotes within this text file method as argument

next one is that to create a DD with collection so here we're having a collection of integers

so well numb is equal to array we're having the set of integers here well numb

odddd is equal to s c. . parallelize num. So, num we have defined here and that num will be passed as input argument to this method

Next one is create RDD from another RDT. So here we are having one collection, array collection

So well num is equal to array and the set of integers we have mentioned

So well, RDT copy is equal to num dot. So this is coming. Num

map now data implies data star 3 so in this way we can create one

odddddd from another odddd next count elements in so in that case well num is equal to array and set up integers are there and then

numb dot count method which will give you the count of elements in the

RDD next one we're going for the filter operation on RD. So, well, text file is equal to SC

. text file and the respective file name with the path will be passed as the input

parameter. So, val temp is equal to text file dot filter. So what is the filtering

criteria? Line implies line. Contents ABC. So all the lines will be filtered, which

will be having the text ABC there. Get first element from RDD. So valtex file is

to s c. text file and then the file path and then text file dot first method so it will

print the first element from the RDD next one is the count of number of partitions in

RDD in that case go for val text file is equal to SC dot text file then give the file with the

path as the input argument enclosed within double quotes and then text file dot partitions

length which will give you the count of number of partitions in RDD

So these are the different set of commands in Spark So let us go for one practical demonstration to show you better that how these commands are working In this demonstration we shall discuss how to execute multiple different Spark commands on our system

So at first, we're going to create one data file. So I'm just going for Walter Control T to open a new terminal here

So in this terminal we'll be working with. now see at first we're going to create one data file the name of the data file will be my

test file dot txt so I'm creating this one so keeping some text there this is a test file

we can create spark rddd with it spark can find number of lines

etc and then we shall go for some symbols so after that control d for save and exit

so let us see that what is the current content of this file so get my test file

txt okay so now I have typed it that is a cat my test file dot txt and we are

getting the file content so now we shall create some RDD so at first we're going to

initiate the spark shell so we know that in that case we're supposed to execute spark

hyphen shell and then the spark shell will get loaded and the scalar prompt

will come so at first we shall create spark rddd using this created data file

so the command will be val my file you can also go for my

file rdd is equal to SC there is a spark context we are going for text file and then we are

giving the path that is slash home slash big data slash my test file dot txt so my test file

txt double code complete and bracket closing so i'm creating i have created my file

RD so this particular RD we are creating from the created data file so in this

we have created now we can go for my file rddd collect then the current content

will be shown in the form of an array you can find that these are respective

array content otherwise there is another very smart way to print it so I shall go

for my file my file rddd dot for each then print ln so it will read all the lines of the file

and they will get printed you see here we're having the respective content

so that is our RDD we have created directly from the file and also we have

printed the file content now let us create one RDD from an array so in that

case the command will be val say my array so I going for this my array is equal to array at first I defining one array here is equal to array say one comma in this way i done how many data we have kept in the array so seven okay so array has got created now we shall

go for say val new rdddd so we are creating one rddd from this array so it will be spark context

dot parallelize my array so we shall go for this new rddd

collect so you can see in the array format it has been shown now also we

can print the same using for each so new rdddd dot forage so new rdddd dot forage so

new rddddd dot for each print elaine so here you can find that they have got printed so all this data they have got

printed in separate lines so that is the array from the array we have created one

RDD okay now copy and update each elements of the array RDD so how to do that

one we shall go for say val update RDD is equal to new rdddd dot map new

RDD dot map the new RDT we created earlier so new RDtt dot map element element and the new

value of the element will be say element say element star say 10 say plus 5 so giving one

RPD expression so now it has got updated so let us go for the print so that is

update RDD so update RDD dot for each print LN so if you go for this so it

will read all the lines of the RDD and they'll get printed you can find

that it has got printed in this way it has got printed in this way why it is

coming as 15 because the first value was one 1 1 in to 10 10 plus 5 so that is a 15

second value was 3 so 3 into 10 30 plus 5 that is a 13

35 the next value was 5 so 5 into 10 50 plus 5 so that is 55 in this way you can find that

this RDT content also we have changed and here you have used the method that is the

respective command will be map so now we shall go for we shall count the number of

elements in the RDD so that can be counted in this way you can go for say new

RDD new RDD is equal to dot count we can call it in this

this way so you can find we are having the seven we had the seven values there so

that's why the value seven is getting printed now we shall go for this my file

rdddd count so we're finding four because we're having the four lines in the

respective file we had the four lines in the respective file so that's why the count is

also showing four here now we shall filter those lines from the file or ddd

where the given word is found so let us go for the filtering so you shall filter those lines from the file or dd where the given word is found so let us go for the filtering so we shall filter those lines from the file rddd where the given word is found so for that one i going for say val

and say filter so going for val filter filter text filter text files so is equal to my file

so my file rdddd dot filter so let us suppose I'm going for each and every line

will get printed and where the line will contain contains the word spark

so just see the command so here we're having say Val filter text file is

equal to my file rdddddd where each and every line those lines will be coming

which line that that very line which is containing the word spark so we have written

so line implies line dot contains spark so that is a command so we are going for

the execution so now let me go for the filter text file filter text file

dot for each print align so if you go on executing we can find that only those

two lines are coming out only those two lines are coming out which is having this

we can create spark RDD with it so spark what what is there and spark and

spark and find number of lines etc so that is the spark what is also there so

in this way we have gone through this so that is the filter text file dot for each

print ellen so in this we can easily find that how we are creating so all these

things so if I go for only for this forage okay I'm going for this I'm going for this as forage so let me see the

file name here so file name is is my file RDD okay so the my file RDD so if I go for

this my file RDD dot forage printallant in that case that you see we're having

these four lines out of these four lines only two lines are having the word spark in it

so that's why those two lines got printed so now if I print the first line of the

text RDD in that case what will happen we shall go for my file rdddd dot fast my file

RDD dot first so I think this particular file will be a f will be in the

capital letter yes so you can find that only the first line is getting printed

so in this way you can find that we can do a multiple different options

operations here using the spark commands we are going for find the size of the text file

partition find the size of the text file partition so let us go for say mine file rdddd

dot partition partition so partitions dot length so you see it is giving that the

partition length is one only so in this way in this demonstration we have shown

that what are the different commands with the help of which we can work with a spark

environment in the scalar prompt thanks for watching this video