Spark Overview

in this video we are discussing spark

overview spark is actually required for

any memory cluster computing and as it

is in memory cluster computing so it is

lightning first and we know that memory

access time would be quite less compared

to the disk accessing time so in case of

in memory cluster computing we have

tried our level best to reduce the

number of disk read and write operations

and also the intermediate results will

be stored in the memory so that's why we

are going to enjoy hundred times faster

in memory cluster computing SPARC is

also having inbuilt API for Scala for

Java and also for Python so SPARC

applications can be developed in

multiple different languages so let us

go for more detailing about this spark

overview so what is SPARC Apache spark

is a lightning-fast cluster computing

technology designed for fast competition

and it is based on Hadoop MapReduce and

the main feature of SPARC is in memory

cluster computing that increases the

processing speed of an application so if

we do the processing doing the read read

operations from the disk obviously that

will be a slower one but here we are

going for in memory cluster computing so

SPARC is one of the Hadoop's sub-project

developed in the year 2009 in UC

berkeley's mplab by metal Johari a--

so here we are discussing the history of

this spark so it was open source in the

year 2010 under the bsd license and it

was donated to a purchase after

foundation in 2013 that's why we call it

as apache spark and now apache spark has

become a top-level Apache project from

the Year February 2014 so there is a

history of apache spark now features of

spark now why the spark has become so

popular let us discuss about its

features first one is the speed spark

helps us to run an application in a

Hadoop cluster

in the or up to 100 times faster in

memory and ten times faster when running

on the disk this is possible by reducing

a number of read/write operations on the

disk because we know that if we reduce

if you can reduce the read/write

operations on the disk the process will

be executing faster that's why we have

done this one in memory and it's towards

the intermediate process to result also

in the memory so when the intermediate

processing will be done the result thus

produced will be also kept in the memory

so supports different languages there is

a next feature so spart provides

building api is in java scala and python

and therefore you can write applications

in different languages as you as well

we're going for the advanced analytics

there is an another very important

feature SPARC not only supports the map

and reduce it also supports SQL queries

and streaming data machine learning and

graph algorithms so that's why we can

develop differing kinds of data analysis

and we can go for advanced analytics

also on this park so let us go for the

installation process how to install

SPARC and Scala on our system we'll be

discussing with one demonstration video

we shall maintain step by step so that

also we can configure your system at

your end so here is the demonstration

for you in this video we are going to

discuss how to install Scala on our

system so here we are having the

download link so that is the HTTP colon

slash slash download slash like been

calm and this way the download link is

there

so we shall copy this download link and

we shall paste it on to our browser so

here we are opening one new tab and then

we have gone for the paste and directly

it will ask for the save the file and

the file will get downloaded onto the

download folder so now the file has got

downloaded automatically so we have

opened the respective zip file so we are

creating one new folder under home so

name of the folder were given a scala so

the folder has got created log into the

folder going for double-click control a

say drag solve the file support copy

download folder has been closed so here

we are having this Scala folder here

we're having this color folder with all

the files extracted now we shall go for

this particular part has to be put in

this / e TC slash profile file so after

downloading the file from the respective

URL so we have extracted the file we

have copied all the contents of the file

and we have pasted it on to a folder

naming the folder as Scala creating a

new folder discolour under the home path

and then we are going for this copy of

this export path is equal to so this

line has to be copied and that has to be

pasted paste it onto this EDC slash

profile file so now we have done the

copy then opening the file so sudo G

edit / EDC slash profile we shall come

at the end of the file and the path has

to be pasted here in for the save and

close now let me execute the profile

file at first so that the path will get

effective now to check whether the Scala

has got loaded or not I'm just going to

execute this color and that is very code

versions I'm going for this so in this

way you have shown you that how to load

scholar onto our system so we're going

for this execution at the Scala prompt

has come in this demonstration we are

going to show you that how to install

spark on our system so at first we are

going to download the spark here from

the link as it has been shown here and

as we are dealing with the Hadoop 2.4 so

the respective 2.4 version of the spark

will be required for our installation

for the compatibility issues so here the

download link for the spark is also

mentioned for the Hadoop 2.4 but also we

can search it from the list of all the

other versions of the spark available so

at first we are going for the copy of

this download link the URL is getting

copied and pasting it here

now he can find that respective version

that is Hadoop 2.4 the spark has to be

downloaded for the Hadoop 2.4 here so

going for the search so these are

download link also mentioned so now we

are going for the two-point full version

of the Hadoop so we have got it clicking

here save the file and then okay so the

file is getting downloaded onto our

download folder so progress has been

observed so the download has been

completed successfully so here we're

having the G file has got extracted

well under the home folder we are going

to create one folder with the name spark

and then log into that folder we're

going for con control a that is a select

all and then we drag all the files onto

the spark folder closing the download

folder here now we are going to set the

respective path here so we are going for

sudo G edit / e TC slash profile so this

particular file has to be updated with

the path giving the password so coming

at the end making a space here so this

the path to be copied so that his export

path is equal to dollar path colon slash

home big data slash spark slash bin so

this respective path has to be put to

that slash ATC slash profile file in the

copy and pasting it at the end saving it

closing it now coming to the terminal we

are going to execute the profile so that

the path will get reflect will get

effective and now we are going for this

spark execution so how to execute this

one spark - shell without having any

blank space in between you can find that

the spark is getting executed and in

this we have shown you that how to

install spark on our system so you can

find the scholar of has come here so

there is a scholar prompt so in this way

you have shown you that how to install

spark in our system and what are the

different steps we require to do so