Some Spark Shell Commands | Spark Commands

In this video we are discussing some spark shell commands

We know that spark is having one interactive shell with a Scala prompt where you can issue

our commands to process our data. So let us go for some further discussion on it

What is the Spark shell? So the Apache Spark is with an interactive shell and it has a Scala prompt and in the

prompt we can run different commands to process our data. spark shell commands will be something like this. So here you have shown some set of very

important spark shell commands. So create a new RDD from the file. So one file is there containing

some data set from there we are going to create one RDD. So well text is equal to SC

dot text file and within this first bracket within the double quotes will be giving the file

path. Here the file path can be the SDFS file path or the local system file path

and this SC actually stands for spark context. So look at the syntax, well text is equal to sc.tects file method, passing this file path within double course as the input parameter

So here the SC is the spark context and in the file path we can provide local or HDFS files

Next one is that create a new RDD from the Scala collection

So we are going to create one adddd from the Scala collection. So here we require to issue two commands The first one well list data is equal to list Here we are having this list of data separated by commas And then well list RDD is equal to SC that is a spark

context, SC.c. Parallelize list data. So this list data will be used here. So now here

we are going to create a new RDD from the Scala collections. Now, count number of items in

RDD. The command will be RDD name dot count method. Next one is that filter operation in

RDD. So, here it will be written as val text is equal to SC dot text file and this is a

text file is a method. So, within first brackets, within double quotes, we are mentioning the

file path and then well, fill data is equal to text.com filter. Text dot filter line implies

line dot contents, you are passing one string there. And depending on

the string depending upon the matching the respective filtering will take place so it will

select those lines which contains a bc string now here we are going for this first item in

rddd the respective command will be text dot first method so in case of select first in items

we are going for the method take so text dot take n so n will be denoting the number of

items we are selecting from the first so in this In this way, we have shown you that what are the different spark shell commands which are

available in our spark environment for the execution in the interactive shell at the

Scala prompt. Thanks for watching this video